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Preface 


This  report  is  a  response  to  Section  1053  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2015  regarding  the  Department  of 
Defense  (DoD)  budget;  the  section  called  for  an  independent  study 
regarding  DoD’s  joint  analytic  capability  and  recommendations  for 
improving  it.  This  report  is  intended  primarily  for  policymakers  and 
staff  in  DoD  and  Congress.  The  report  may  also  be  helpful  to  the 
much  broader  community  interested  in  analysis  for  strategic  planning, 
whether  in  government  or  the  private  sector,  and  related  methods  and 
tools.  Comments  and  questions  are  welcome  and  can  be  addressed  to 
the  author  at  pdavis@rand.org. 

The  research  for  this  report  was  conducted  for  the  Office  of  the 
Secretary  of  Defense  within  the  International  Security  and  Defense 
Policy  Center  of  the  RAND  National  Defense  Research  Institute,  a 
federally  funded  research  and  development  center  sponsored  by  the 
Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the  Unified  Com¬ 
batant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agencies, 
and  the  defense  Intelligence  Community. 

For  more  information  on  the  International  Security  and  Defense 
Policy  Center,  see  www.rand.org/nsrd/ndri/centers/isdp  or  contact  the 
director  (contact  information  is  provided  on  web  page). 
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Summary 


Background 

The  U.S.  Department  of  Defense  (DoD)  has  long  had  broad  capabilities 
for  joint  analysis,  whether  for  planning  future  forces  or  supporting  field 
commanders  during  current  challenges.  DoD  is  envied  by  other  govern¬ 
ment  agencies  for  its  analytic  capabilities  and  processes.  Nonetheless, 
some  issues  have  arisen.  This  report  stems  from  a  congressional  request 
for  an  independent  report  about  DoD’s  capabilities  for  joint  analysis 
and  ways  to  improve  them.  Congressional  concerns  largely  involved  the 
activity  called  support  for  strategic  analysis  (SSA).  In  2011,  the  direc¬ 
tor  of  the  Office  of  the  Secretary  of  Defense’s  (OSD’s)  Cost  Assessment 
and  Program  Evaluation  (CAPE)  disbanded  campaign-modeling  and 
reduced  CAPE’s  participation  in  the  SSA  activity,  weakening  SSA  sig¬ 
nificantly.  This  report  is  largely  about  SSA  and  how  and  whether  to 
revise  it. 


Differing  Views 

CAPE’s  decision  reflected  a  conclusion,  accepted  by  the  Secretary  of 
Defense  and  some  other  senior  leaders,  that  the  SSA  process  had  not 
helped  decisionmakers  confront  their  most-difficult  problems.  The 
activity  had  previously  been  criticized  for  having  been  mired  in  tra¬ 
ditional  analysis  of  kinetic  wars  rather  than  counterterrorism,  inter¬ 
vention,  and  other  “soft”  problems.  The  actual  criticism  was  broader: 
Critics  found  SSA’s  traditional  analysis  to  be  slow,  manpower-inten- 
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sive,  opaque,  difficult  to  explain  because  of  its  dependence  on  com¬ 
plex  models,  inflexible,  and  weak  in  dealing  with  uncertainty.  They 
also  concluded  that  SSA’s  campaign-analysis  focus  was  distracting 
from  more-pressing  issues  requiring  mission-level  analysis  (e.g.,  how  to 
defeat  or  avoid  integrated  air  defenses,  how  to  defend  aircraft  carriers, 
and  how  to  secure  nuclear  weapons  in  a  chaotic  situation).  CAPE  felt 
that  the  focus  on  analytic  baselines  was  reducing  its  ability  to  provide 
independent  analysis  to  the  secretary.  The  campaign-modeling  activ¬ 
ity  was  disbanded,  and  CAPE  stopped  developing  the  corresponding 
detailed  analytic  baselines  that  illustrated,  in  detail,  how  forces  could 
be  employed  to  execute  a  defense-planning  scenario  that  represented 
strategy.  During  the  secretary’s  reviews  for  fiscal  years  2012  and  2014, 
CAPE  instead  used  extrapolated  versions  of  combatant  commander 
plans  as  a  starting  point  for  evaluating  strategy  and  programs. 

The  CAPE  decision  was  controversial.  The  services  all  formally 
expressed  their  concerns  because  they  value  the  SSA  process  and  believe 
that  missing  aspects  of  SSA  should  be  restored  so  that  joint  analytic 
baselines  will  be  available  to  use  for  the  services’  analyses  (e.g.,  analysis 
of  alternatives  [AoAs]  and  program  development  over  time).  The  deci¬ 
sion  was  also  controversial  within  OSD.  Some  senior  officials  believed 
from  personal  experience  that  SSA  had  been  very  useful  for  behind- 
the-scenes  infrastructure  (e.g.,  a  source  of  expertise  and  analytic  capa¬ 
bility)  and  essential  for  supporting  DoD’s  strategic  planning  (i.e.,  in 
assessing  the  executability  of  force-sizing  strategy).  These  officials  saw 
the  loss  of  joint  campaign-analysis  capability  as  hindering  the  ability 
and  willingness  of  the  services  to  work  jointly.  The  officials  also  dis¬ 
agreed  with  using  combatant  commander  plans  instead  of  scenarios  as 
starting  points  for  review  of  midterm  programs,  because  such  plans  are 
too  strongly  tied  to  present-day  thinking. 

The  reader  will  appreciate,  then,  that  disagreements  are  strong 
across  and  even  within  DoD  offices.  The  conclusions  and  recommen¬ 
dations  on  the  future  of  SSA  are  based  on  interviewing,  selective  review 
of  government  documents,  and  prior  research.  Although  idealized,  the 
proposed  approach  is  feasible.  It  builds  on  the  many  successes  of  the 
past  SSA  and  on  modern  methods  of  analysis. 
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Conclusions  and  Recommendations 

Research  for  this  report  substantiated  both  the  criticisms  that  led  to 
CAPE’s  actions  and  the  need  to  recreate  capabilities  that  have  been  lost, 
albeit  in  different  form.  What  should  be  done,  however,  is  not  straight¬ 
forward,  and  halfway  measures  could  be  counterproductive. 

Background  for  Recommendations:  Reconceiving  the  Larger 
Planning  Construct 

Some  of  the  root  problems  affecting  SSA  stem  from  how  DoD  con¬ 
ceived  SSA  years  ago.  Figure  S.l  is  a  simplified  version  of  the  approach 
as  it  existed  through  2011  for  midterm  planning. 

At  first  glance,  this  pre-2012  approach  seems  reasonable.  In  this 
construct,  the  Office  of  the  Under  Secretary  of  Defense  for  Policy 
(OUSDP)  issues  the  defense-planning  scenarios  based  on  defense  strat¬ 
egy,  the  Joint  Staff  specifies  concepts  of  operations  (CONOPS),  and 
CAPE  then  creates  a  study  to  specify  enough  contextual  detail  to  jus¬ 
tify  assumptions  and  other  data  necessary  to  feed  campaign  models. 
CAPE  reports  the  package  as  an  analytic  baseline  consisting  of  a  sce¬ 
nario  (a  written  description),  CONOPS,  integrated  joint  data,  and 
model  results.  This  baseline  allows  competing  analyses  to  be  compared 
with  common  data  and  assumptions,  thereby  making  evaluation  easier. 
The  baseline  also  provides  initial  data  for  subsequent  analyses,  such  as 
AoAs  and  joint  studies. 

Upon  reflection,  the  larger  construct  of  Figure  S.l  has  serious 
problems.  First,  it  omits  the  most  important  step:  determining  the 
defense  strategy  and  budget  in  the  first  place.  Does  that  not  need 
resource-informed  analysis  (in  addition  to  background  research)? 
Second,  the  figure  shows  planning  scenarios  flowing  out  of  strategy. 
Does  that  not  need  analysis?  Third,  in  practice,  the  CONOPS  devel¬ 
oped  to  execute  the  scenarios  end  up  specifying  a  much  more  pre¬ 
cise  scenario  than  intended  by  the  descriptive  scenarios  generated  by 
OUSDP.  Whereas  OUSDP’s  scenarios  depict  numerous  difficulties, 
dilemmas,  and  uncertainties — sketching  a  rich  problem  space — the 
CONOPS  are  developed  to  describe  a  narrow  path  through  the  space. 


Figure  S.1 

Simplified  Depiction  of  the  Support  for  Strategic  Analysis  Process  (Through  2011)  for  Midterm  Planning 


Other  information 

NOTE:  PPBE  =  planning,  programming,  budgeting,  and  executing;  SecDef  =  Secretary  of  Defense;  AoA  =  analysis  of  alternatives. 
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Although  some  excursions  can  be  performed,  many  crucial  uncertain¬ 
ties  and  disagreements  are  lost. 

This  lack  of  excursions  was  not  intended:  Analytic  baselines  were 
to  be  mere  points  of  departure  for  far-reaching  excursions,  but  except 
for  some  included  with  the  baseline  packages,  such  excursions  proved 
rare  because — given  SSA  methods — they  required  new  CONOPS 
and  other  new  data,  the  development  and  coordination  of  which  were 
a  major  undertaking.  The  resulting  analytic  process  buried  uncer¬ 
tainty  and  disagreement.  The  process  was  unable  to  adequately  assist 
senior  leaders  when  they  sought  to  address  uncertainty,  what-ifs,  and 
trade-offs.  The  process  was  also  lengthy  and  tedious  despite  admirable 
cooperation,  expertise,  and  professionalism. 

Ironically,  the  Secretary  of  Defense  introduced  capabilities-based 
planning  in  2001,  after  a  decade  of  recommendations  about  dealing 
better  with  uncertainty.  Successive  secretaries  have  reinforced  that 
intent,  which  translates  into  a  desire  to  seek  capabilities  providing  flex¬ 
ibility,  adaptiveness,  and  robustness  (FARness)  to  the  extent  permit¬ 
ted  by  a  budget.1  The  analytic  process,  however,  has  seriously  undercut 
such  intentions.  A  root  problem  has  been  that  the  level  of  strategic  analysis 
has  been  too  detailed,  precluding  more  far-reaching  analysis.  That  detail 
has  also  generated  objections  from  the  Joint  Staff  and  services,  who  feel 
that  the  baselines  have  overly  specified  crucial  matters  that  should  be 
addressed  contextually  by  the  military. 

Another  objection  to  the  SSA  activity,  both  before  and  after  2011, 
is  that  it  has  accepted  uncritically  the  services’  programs  of  record  and 
traditional  CONOPS.  This  is  a  particular  problem  when  defense  sec¬ 
retaries  are  deeply  concerned  about  such  adverse  trends  as  increasingly 
capable  adversaries  with  better  operational  concepts  and  seek  to  take 
advantage  of  new  technologies  with  innovative  concepts  of  organiza¬ 
tion  and  operations,  rather  than  rely  on  extrapolations  from  the  past. 

A  final  background  observation:  The  pre-2012  SSA  activity  was 
conceived  of  as  providing  standardized  models  and  data  (including 
assumptions)  for  subsequent  detailed  campaign  analysis.  Common 
joint  data  on  factual  matters  are  important,  and  having  a  reference 
set  of  common  assumptions  is  useful  when  comparing  analyses,  but 
the  near-exclusive  de  facto  focus  on  standard  models  and  standard 


xiv  Capabilities  for  Joint  Analysis  in  the  Department  of  Defense 


inputs  has  been  counterproductive,  defeating  the  intentions  of  plan¬ 
ning  under  uncertainty.  The  pre-2012  activity  created  substantial  over¬ 
head  expense,  with  questionable  analytic  value  to  leadership.  Revital¬ 
ized  analytic  capability  is  needed,  but — if  the  activity  is  to  be  more 
useful — DoD  should  establish  a  base  of  uncertainty-sensitive  analy¬ 
sis  on  which  decision-oriented  analysis  can  draw  when  necessary — 
e.g.,  well-conceived  trade-off  charts  relating  both  to  objectives  and  to 
risks.  The  recommendations  described  below  would  accomplish  that, 
among  other  things. 

A  New  Planning  Construct:  Analytic  Support  for  Strategic  Planning 

Against  this  background,  this  report  recommends  that  DoD  should 
create  a  new  activity  to  replace  SSA.  The  new  activity  could  be  called 
analytic  support  for  strategic  planning  (ASSP). 

1.  ASSP  should  support  initial,  interim  decisions  on  defense  strat¬ 
egy  and  budgets  with  explicit,  understandable,  and  therefore 
relatively  simple  analysis. 

2.  ASSP  should  emphasize  planning  for  flexibility,  adaptiveness, 
and  robustness  (FARness);  it  should  deemphasize  detailed  ana¬ 
lytic  baselines.  As  occurs  now,  it  should  develop  a  list  of  type 
scenarios  (e.g.,  defeat  a  particular  rogue  state  that  is  attacking 
an  ally,  prevail  in  two  simultaneous  wars,  and  cope  adequately 
with  even  more-complex  cases).2  Then,  for  each,  ASSP  should 
develop  spanning  sets  of  variations  to  stress  U.S.  capabilities  in 
all  the  dimensions  needed.  The  spanning-set  scenarios  should 
constitute  necessary  and  sufficient  requirements  for  the  services 
to  meet  in  their  program  development. 

3.  ASSP  should  include  options  that  incorporate  emerging  technol¬ 
ogy  and  innovative  concepts.  To  assist  in  doing  so,  DoD  should 
elevate  the  role  of  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  (USD[AT&L])  and 
have  the  Joint  Staff  be  more  active  in  assuring  the  surfacing  of 
innovative  service  options  (not  just  programs  of  record  and  tra¬ 
ditional  CONOPS). 
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4.  ASSP  should  help  OSD  focus  on  strategic  considerations  and 
relatively  low-resolution  analysis  when  establishing  planning 
scenarios  or  studying  mission-level  issues  (also  called  capability 
issues),  with  the  Joint  Staff  having  the  primary  role  for  the  next 
level  of  detail  that  is  especially  important  to  the  services. 

5.  DoD  should  require  the  Joint  Staff  to  be  more  active  in  review¬ 
ing  and  critiquing  service  programs  and  requirement  estimates. 

6.  ASSP  should  change  the  mix  of  analytic  methods,  increasing 
the  emphasis  on  lower-resolution  analysis  with  relatively  simple 
qualitative  models,  quantitative  models,  and  human  wargam¬ 
ing.  These  should  be  sound  abstractions  from  more-detailed 
(higher-resolution)  work.  ASSP  should  apply  this  mixed-method 
approach  to  both  campaign-  and  mission-level  analyses. 

Figure  S.2  schematically  depicts  this  construct  for  mid-  and 
longer-term  planning.  The  lightly  and  more  darkly  shaded  portions 
indicate  what  would  fall  within  ASSP  (with  the  option  of  considering 
the  activity  of  the  colored  oval  as  separate,  since  it  is  accomplished  by 
OSD  and  the  Joint  Staff  alone).  As  discussed  more  fully  in  the  text, 
implementation  would  require  significant  and  difficult  changes  of  orga¬ 
nization,  process,  culture,  methods,  and  skill  mix. 

Early  Decisions  with  Scenario  Sets  as  Outputs 

The  first  step  in  Figure  S.2  is  to  inform  initial  decisions  about  strategy 
and  budget.  The  primary  issue  is  what  challenges  the  United  States 
will  take  on  in  developing  its  defense  program — for  example,  does  the 
country  want  the  ability  to  fight  and  win  two  near-simultaneous  wars 
with  regional  adversaries,  that  plus  deterring  several  others,  or  some¬ 
thing  more  complex?  The  more  challenges  the  United  States  takes  on, 
and  the  higher  the  goals  set  for  each,  the  higher  the  budget  that  is 
necessary.  Some  goals  are  implausible  even  with  very  high  budgets. 
Informing  such  decisions  requires  analysis  that  is  understandable, 
broad  in  scope,  and  helpful  in  making  strategic  choices  despite  deep 
uncertainty.  Choices  need  to  strike  a  balance  across  objectives  and  risks. 

If  the  decisions  are  to  determine  the  requirements  for  the  mili¬ 
tary  services,  then  what  requirements  are  needed,  and  how  should 


Figure  S.2 

A  Process  for  Analytic  Support  for  Strategic  Planning  (ASSP) 


Analytic  feedback  for  next  cycle  (e.g.,  revised  capability  plots  and  risk  assessment) 
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NOTES:  Light  and  dark  gray  areas  indicate  the  domain  of  ASSP;  work  in  the  darker  area  is  somewhat  more  detailed.  Policy  =  Office  of 
the  Under  Secretary  of  Defense  for  Policy;  PPBE  =  planning,  programming,  budgeting,  and  executing;  SecDef  =  Secretary  of  Defense. 
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they  be  expressed?  Using  concrete  scenarios  is  useful  and  traditional 
for  DoD.  The  issue  is  deciding  which  and  what  kind.  A  major  change 
in  the  1990s  was  DoD  beginning  to  use  a  number  of  type  scenarios 
(e.g.,  defeating  a  regional  rogue,  defeating  a  near-peer  regional  power, 
and  simultaneous  conflicts).  DoD  has  since  added  more-complex  com¬ 
bination  challenges  (e.g.,  dealing  with  attacks  on  the  homeland  while 
being  engaged  abroad  in  two  theaters).  Each  type  scenario  could  arise 
in  diverse  ways,  demanding  different  capabilities  of  U.S.  forces.  A  prin¬ 
ciple  for  evaluating  a  force-planning  strategy  is  that  analysis  must  cover 
the  variations:  It  is  having  the  capability  to  do  well  across  the  range  of 
reasonable  possibilities  that  matters,  not  just  having  the  capability  for 
some  base  cases. 

As  demonstrated  in  modern  work,  dealing  with  myriad  variations 
coherently  can  be  accomplished  using  a  well-chosen  spanning  set  of 
scenarios;  capability  to  deal  with  those  will  imply  capability  to  deal 
with  in-between  or  lesser-included  possibilities.  This  is  not  planning 
for  the  so-called  worst  case,  because  no  such  worst  case  exists.  A  case 
that  is  the  worst  in  some  respects  (e.g.,  short  warning)  will  be  easy  in 
others  (e.g.,  the  adversary  will  also  have  fewer  forces  than  after  a  mobi¬ 
lization).  Consequently,  if  requirements  are  to  be  expressed  as  test-case 
scenarios,  many  such  cases  (a  spanning  set)  are  needed  for  each  type 
scenario.  Defining  these  is  straightforward  for  analysts,  but  not  so  for 
committees.  Systematizing  such  work  will  be  a  natural  but  important 
extension  of  the  analysis  for  the  2010  Quadrennial  Defense  Review, 
which  developed  different  spanning  sets  to  test  the  overall  force  struc¬ 
ture  by  stressing  the  forces  in  different  ways. 

Contrasting  Old  and  New  Approaches 

Figure  S.3  contrasts  old  and  new  approaches.  Under  SSA,  an  analytic 
baseline  was  a  single  rather  detailed  example  (the  ovals)  of  a  type  sce¬ 
nario  or  challenge.  In  the  new  approach,  each  such  challenge  type  is 
represented  by  a  set  of  low-resolution  cases  (the  circles)  presenting  differ¬ 
ent  stresses  for  U.S.  capabilities.3  Because  of  the  simple  low-resolution 
character  of  the  cases,  it  is  possible  with  suitable  models  to  examine 
analytically  all  of  the  cases  and  to  include  parametric  variations.  As 
illustrated  schematically  on  the  right  side  of  Figure  S.3,  results  of  such 
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Figure  S.3 

Single  Type  Scenarios,  Spanning  Sets,  and  Capability  Maps 


Challenge  Class 

Defeat  regional 
rogue 


Old:  Detailed 
Analytic-Baseline 
Cases 


Defeat  near-peer 
regional  power 


Intervene  as 
peacemaker 


Defend  in  two  theaters, 
recover  from  event 
in  homeland,  and  .  .  . 


Capability  Map 
(a  better  analystic 
baseline) 


O  O  O  O.  .  .  Adversary 

strength 
(size,  quality) 

o  o  o  o... 

Preparation  time 

o  o  o  o  o  o... 


New:  Analytically 
Derived  Low-Resolution 
Spanning  Sets  of  Cases 


o  o  o  o.. 


NOTES:  Green,  yellow,  and  red  correspond,  respectively,  to  probable  success, 
uncertain  results,  or  probable  failure.  Bullets  indicate  test  cases,  which  constitute 
the  spanning  set. 
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exploratory  analysis  can  be  presented  in  capability  maps  showing  for 
which  cases  capabilities  are  sufficient,  adequate  for  deterrence  at  best, 
or  inadequate.  That  is,  green,  yellow,  and  red  correspond,  respectively, 
to  probable  success,  uncertain  results,  or  probable  failure.  Good  quan¬ 
titative  analysts  can  develop  these  or  analogous  maps  based  on  broad 
considerations  and  low-resolution  models;  the  analysts  can  then  identify 
the  test  cases  (indicated  by  bullets)  that  stress  the  future  force  in  the  dif¬ 
ferent  ways  needed  and  that  constitute  the  spanning  set.  Those  sets  can 
be  studied  in  more  detail,  improving  and  enriching  the  capability  maps. 
These  selective,  detailed  looks  are  sometimes  crucial. 

Why  is  this  important?  Why  not  just  have  one  illustrative  sce¬ 
nario  of  each  class  and  assume  that  the  services  will  build  capabilities 
for  all  the  variants?  Arguably,  during  the  Cold  War,  U.S.  defense  bud¬ 
gets  were  large,  prices  were  lower,  and  the  services  included  just-in-case 
capabilities  and  slack.  That  is  no  longer  the  case  given  the  budgetary 
strains  on  the  U.S.  military  as  noted  with  alarm  in  the  2014  Quadren- 
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nial  Defense  Review.  DoD  needs  to  be  more  explicit  about  the  breadth 
of  requirements,  or  the  capabilities  might  not  be  programmed. 

Although  the  colored  capability  map  in  Figure  S.3  is  simplified 
and  has  only  two  dimensions  of  uncertainty  (preparation  time  and 
adversary  strength),  the  basis  for  more-complex  versions  of  such  analy¬ 
sis  has  been  laid  over  the  past  two  decades  with  concrete  examples. 
The  approach  is  not  hypothetical  and  can  be  comprehensibly  extended 
to  more  low-resolution  dimensions  of  uncertainty  (e.g.,  aggregate-level 
characterizations  of  the  behavior  of  U.S.  allies  and  adversary  allies,  the 
operational  strategies  adopted,  the  effectiveness  of  new  weapon  sys¬ 
tems,  and  the  effect  of  such  shocks  as  initial  cyber  attacks  on  command 
and  control).  Although  now  well  demonstrated,  such  analysis  requires 
different  methods  and  models  than  those  used  in  the  SSA  activity,  as 
well  as  people  skilled  in  using  the  new  methods  and  models. 

Balancing  Mission-  and  Campaign-Level  Analyses 

Another  objective  for  an  ASSP  is  elevating  the  relative  emphasis  on 
mission-level  work  (also  called  capability-area  work).  Many  cru¬ 
cial  defense-planning  issues  are  about  how  to  achieve  capabilities  for 
accomplishing  such  increasingly  difficult  missions  as  defeating  inte¬ 
grated  air  defenses.  The  ASSP  activity  should  achieve  a  balance  that 
gives  at  least  as  much,  if  not  more,  attention  to  mission-level  issues. 
OUSDP  and  CAPE  should  be  concerned  primarily  with  higher-level 
(lower-resolution)  aspects  of  both,  while  the  services  must  address  such 
matters  in  more  depth.  Precisely  the  same  methods  as  described  above 
apply  to  mission-level  analysis. 

Despite  the  challenges  in  doing  so,  ASSP  should  also  achieve  a 
balance  with  respect  to  types  of  conflict.  In  particular,  ASSP  should 
give  appropriate  weight  to  analysis  relating  to  counterterrorism,  stabili¬ 
zation,  and  irregular  warfare.  The  methods  for  studying  these  matters 
should  draw  on  social  science  (and,  in  some  cases,  be  the  methods  of 
social  science). 

Responsibilities 

As  summarized  in  Table  S.l,  the  new  construct  indicated  in  Figure  S.2 
changes  the  responsibilities  of  DoD  components  in  important  ways. 
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Table  S.1 

Responsibilities  in  the  Analysis  for  Support  of  Strategic  Planning  Activity 


Inform  Initial 
Decisions  on 
Strategy  and 
Budget  (Task 
Force  Co-Led 
by  Cape  and 
OUSDP) 

Articulate 
Strategy, 
Challenges, 
and  Spanning 
Sets  for 
Challenge 

Refine 

Strategic 

CONOPS 

Refine 

Strategic 

Capability 

Requirements 

and 

Assessments 
of  Capability 
and  Risk 

Refine 

Detailed 

Requirements 

and 

Assessments 
of  Capability 
and  Risk  as 
Needed 

OUSDP 

•  ••• 

•  ••• 

• 

• 

• 

Joint  Staff 

•  • 

• 

•  ••• 

•  • 

•  ••• 

CAPE 

•  ••• 

• 

•  • 

•  ••• 

•  • 

USD(AT&L) 

•  • 

• 

•  • 

•  • 

•  • 

SecDef 

VCJS,  J5, 
or  J8 

Approval 

Approval 

Approval 

Approval 

Approval 

NOTES:  The  task  force  activity  (left  side)  might  or  might  not  be  considered  part  of 
ASSP  per  se.  Number  of  bullets  indicates  relative  responsibility.  SecDef  =  Secretary 
of  Defense;  VCJS  =  Vice  Chairman,  Joint  Staff;  J5  =  Director  for  Strategic  Plans  and 
Policy,  Joint  Staff;  J8  =  Director,  Force  Structure,  Resources,  and  Assessment,  Joint 
Staff. 


The  first  step  of  developing  defense-planning  scenarios  is  replaced  by 
the  larger  step  of  informing  interim  decisions  on  strategy  and  budget. 
This  should  be  accomplished  by  an  elite  team  or  task  force  (not  a  com¬ 
mittee)  led  jointly  by  CAPE  and  OUSDP — CAPE  because  the  effort 
requires  rigorous  (albeit  low-resolution)  analysis  involving  capabilities, 
costs,  and  trade-offs  and  OUSDP  because  the  effort  is  about  establish¬ 
ing  strategy.  DoD  should  consider  having  some  top-quality  analysts 
functionally  dual-hatted  for  CAPE  and  OUSDP  (even  though  the  ana¬ 
lysts  would  formally  reside  in  only  one).  DoD  used  an  approach  of 
this  sort  in  earlier  decades.  Representatives  from  USD(AT&L)  and  the 
Joint  Staff  should  ensure  attention  to  innovation  and  military  feasibil¬ 
ity.  Results  should  be  approved  by  the  Secretary  of  Defense. 

The  next  step  shown  in  the  table  is  that  OUSDP  and  the  Joint 
Staff  should,  as  before,  articulate  strategy  and  develop  CONOPS, 
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respectively.  The  last  two  columns  are  new,  corresponding  to  the  lightly 
and  more  darkly  shaded  regions  on  the  right  side  of  Figure  S.2.  In  this 
construct,  the  more  strategic,  low-resolution  activities  are  primarily 
the  responsibility  of  CAPE,  with  the  Secretary  of  Defense  approving 
the  final  expression  of  requirements.  Those  of  a  more  detailed  nature 
are  primarily  the  responsibility  of  the  Joint  Staff.  Although  all  ASSP 
contributors  would  review  and  comment  on  everything,  as  occurs 
within  SSA,  Table  S.l  indicates  that  the  Joint  Staff  and  USD(AT&L) 
should  be  involved  in  the  low-resolution  refining  and  that  CAPE  and 
USD(AT&L)  should  be  involved  in  the  more  detailed  work.  The  goal 
should  not  be  consensus  but  ensuring  quality  and  innovation. 

The  later  portions  of  the  ASSP  process  (the  part  most  akin  to  the 
current  SSA)  would  have  new  responsibilities  for  refining  capability 
maps  and  related  risk  assessments.  Although  ASSP  would  do  support¬ 
ing  analysis  (e.g.,  the  creation  of  capability  maps),  the  analysis  directly 
aiding  decisionmaking  would  continue  to  depend  on  CAPE  (for  the 
Secretary  of  Defense)  and  the  various  special  groups  serving  the  chair¬ 
man  and  service  chiefs.  ASSP  would  still  be  providing  support,  not 
final  analysis,  but  its  products  would  be  far  more  informative  than  data 
for  point  cases. 

In  this  construct,  CAPE  would  reassume  some  of  the  role  it  had  in 
SSA,  but  with  crucial  differences.  CAPE  would  focus  primarily  on  the 
strategic  level  rather  than  be  dragged  into  more  detail,  and  it  would  be 
refining  higher  capability  requirements  at  campaign  and  mission  levels 
rather  than  specifying  the  assumptions  of  a  standard  case  of  a  detailed 
campaign  model.  CAPE  would  be  involved  to  some  extent  in  details, 
but  in  connection  with  capability  requirements  and  risk  assessment, 
not  creating  data  for  big  models.  The  Joint  Staff  would  be  expected 
to  be  a  stronger  advocate  for  innovation  and  a  more  critical  reviewer 
of  service  requests.  USD(AT&L)  would  advocate  for  innovative  and 
sometimes  disruptive  options  that  could  affect  the  size  and  character 
of  the  force.  These  are  all  nontrivial  changes  essential  to  ensuring  that 
ASSP  is  a  meaningful  advancement  on  the  preexisting  SSA.  Some  of 
the  important  changes  will  not  occur  without  strong  influence  from 
the  secretary  (hence,  some  readers  will  be  skeptical). 
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Analytic  Methods 

The  revised  activity  described  above  is  impossible  without  a  change  in 
analytic  methods.  ASSP  requires  an  appropriate  mix  of  methods  and 
tools  and  the  means  by  which  to  integrate  across  types  of  knowledge 
and  levels  of  resolution.  Such  integration  has  precedent  but  is  rare,  intel¬ 
lectually  difficult,  and  easy  to  do  poorly.  Further  theoretical  advances 
and  related  tools  are  needed.  Nonetheless,  much  can  be  done  now  by 
having  the  right  people  assigned  to  the  effort  so  that  technical,  tactical, 
operational,  and  strategic-level  knowledge  is  mutually  informed. 

The  new  mix  of  methods  should  include  campaign-  and  mission- 
level  analysis  approached  at  various  levels  of  detail  (multiresolution 
modeling).  Initial  priority  should  be  on  finding  or  building  relatively 
low-resolution  models  that  are  suitably  parameterized. 

Staffing 

Staffing  should  be  approached  by  asking  what  mix  of  talents  and  expe¬ 
riences  is  needed  and  what  mix  is  appropriate  for  work  within  the  gov¬ 
ernment,  in  federally  funded  research  and  development  centers,  or 
industry.  As  in  the  past,  the  best  talent  may  have  disciplinary  back¬ 
grounds  in  the  physical  sciences,  engineering,  mathematics,  econom¬ 
ics,  and  other  relevant  social  sciences.  ITiring  standards  should  not 
associate  the  title  analyst  with  a  particular  discipline.  One  objective  in 
rebalancing  should  be  to  attract  staff  capable  of  higher-level  modeling 
and  analysis,  crosscutting,  and  integration. 

Research 

Solid  analysis  requires  a  basis  in  reality.  The  new  ASSP  activity  should 
have  a  research  component  to  ensure  a  continuing  stream  of  up-to-date 
knowledge  and  capability,  as  well  as  an  active  effort  to  obtain  deep 
knowledge  about  the  most-crucial  subjects  (e.g.,  counterterrorism,  sur¬ 
vivability  of  small  high-technology  ground-force  units,  robotics,  and 
cyber  war).  Elements  should  include  the  following: 

•  Research  funds:  Draw  knowledge  from  such  diverse  sources  as 
history,  operational  experience,  field  experiments,  operational 
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data,  experience  of  foreign  militaries,  and  the  social  sciences  and 
then  integrate  knowledge. 

•  Investment  in  new  qualitative  and  quantitative  analytic 
methods  and  tools,  including  human  gaming:  Put  priority 
on  relatively  simple  capabilities  that  deal  well  with  uncertainty 
and  make  key  assumptions  and  their  significance  clear,  on  meth¬ 
ods  that  exploit  social  science  knowledge  where  relevant,  and  on 
methods  that  integrate  across  such  techniques  as  human  wargam¬ 
ing  and  modeling.  Support  work  at  multiple  levels  of  resolution 
and  require  modularity  to  enable  sharing,  competition,  and  com¬ 
position  in  future  campaign  models.  DoD  should  not  contem¬ 
plate  a  new,  complex,  and  monolithic  simulation. 

•  Better  leveraging  of  professional  societies:  Staffs  benefit  greatly 
from  professional  activities,  and  DoD  should  encourage  participa¬ 
tion  and  request  the  professional  societies  to  have  more  in-depth 
meetings  and  written  papers.  Often,  DoD  should  commission 
follow-up  activities  to  generate,  for  example,  tightly  constructed 
definitions  disambiguating  broad  guidance  and  reconciling  dif¬ 
ferent  services’  understanding  of  terms. 


Caveat:  Beware  of  Halfway  Measures 

If  this  report’s  recommendations  are  accepted  nominally  but  imple¬ 
mentation  fails  to  achieve  changes  in  culture,  methods,  and  staff  mix, 
the  result  might  be  worse  than  the  present.  ASSP  might  get  in  the  way 
of  the  analysis  and  decision-aiding  that  now  goes  on  from  non-SSA 
activities  in  CAPE,  OUSDP,  USD(AT&L),  and  the  Joint  Staff.  Merely 
revitalizing  SSA  by  restoring  past  capabilities  and  methods  would  be 
recreating  a  system  already  known  to  be  inadequate. 


Next  Steps 

This  report  says  relatively  little  about  implementation,  which  should 
depend  on  many  other  considerations,  such  as  the  preferences  of  the 
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Secretary  of  Defense.  A  new  administration  will  be  taking  office  in 
2017,  so  preferences  could  change.  A  good  deal  of  groundwork  should 
be  laid  in  the  meantime. 

•  DoD  should  commission  a  background  report  to  describe  in  more 
detail  what  would  constitute  a  coherent  approach  to  joint  analysis 
that  includes  low-  and  medium-resolution  modeling  and  human 
wargaming,  integration  of  knowledge,  and  supportive  research. 
This  would  reflect  an  assessment  of  which  elements  are  already  in 
place,  which  are  available  but  not  in  place,  and  which  would  need 
development.  The  report  would  include  definitions,  examples, 
and  technical  discussion. 

•  DoD  should  commission  a  campaign  of  briefings  and  workshops 
to  acquaint  the  several  relevant  communities  (analysts,  decision¬ 
makers,  etc.)  with  the  ideas  proposed  and  to  invite  discussion  and 
debate.  These  briefings  and  workshops  should  include  outreach  to 
the  larger  interested  community  beyond  DoD. 

•  DoD  should  identify  alternative  implementation  strategies, 
including  some  human  gaming  of  strategies  to  better  understand 
the  potential  consequences,  both  good  and  bad.  As  part  of  this, 
DoD  should  identify  sensible  time-phasing.  It  will  not  be  possible 
to  implement  all  the  suggestions  of  this  report  in  a  short  period 
(e.g.,  establishing  multiresolution  families  of  models).  A  two-year 
plan  would  make  sense,  with  important  changes  occurring  early 
and  others  requiring  more  time. 

In  contrast  with  the  current  report,  which  was  mandated  as  an  inde¬ 
pendent  study,  this  groundwork-laying  should  involve  those  currently 
involved  in  DoD  analysis  and  the  current  SSA.  The  next  study  will 
need  support  and  direction  from  the  secretary  or  deputy  secretary. 
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CHAPTER  ONE 


Introduction 


Objectives  and  Background 

The  objective  of  this  report  is  to  provide  an  “independent  assessment 
of  the  joint  analytic  capabilities  of  the  Department  of  Defense  [DoD] 
to  support  strategy,  plans,  and  force  development  and  their  link  to 
resource  decisions,”  as  called  for  in  the  National  Defense  Authoriza¬ 
tion  Act  for  Fiscal  Year  2015  (U.S.  Congress,  2014;  see  Appendix  A). 

The  report  came  about  because  Congress  became  aware  of  sev¬ 
eral  concerns  relating  to  DoD’s  joint  analysis.  These  concerns  primarily 
involved  the  support  for  strategic  analysis  (SSA)  activity,  which  is  the 
main  focus  of  this  report.  In  2011,  the  director  of  the  Office  of  the  Sec¬ 
retary  of  Defense’s  (OSD’s)  Cost  Assessment  and  Program  Evaluation 
(CAPE)  disbanded  campaign-modeling  and  reduced  CAPE’s  partici¬ 
pation  in  the  SSA  activity,  weakening  SSA  significantly.  The  legislation 
asked  for  recommendations  about  capability  improvements  and  related 
changes  of  process  and  organization. 

Information-Gathering 

The  first  research  phase  involved  gathering  information  over  several 
months  by  in-person  interviewing  of  present  and  past  DoD  officials 
and  working  analysts.  I  collected  concerns,  perspectives,  and  sugges¬ 
tions  on  a  not-for- attribution  basis  to  encourage  candor.1  I  also  exam¬ 
ined  selected  interim  and  final  written  materials  to  gain  an  updated 
understanding  of  SSA  and  its  products.2 

A  major  event  affecting  the  views  of  all  of  those  interviewed  was 
the  decision  by  the  director  of  CAPE  in  2011  to  cease  developing  SSA’s 
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analytic  baselines  and  to  disband  a  sizable  group  charged  with  cam¬ 
paign-modeling.3  The  decision  was  controversial  and  had  ramifications 
that  persist  to  this  day  and  motivated  the  congressional  request  for  this 
report. 

The  concerns  expressed  by  interviewees  fell  into  two  groups:  those 
seeing  great  value  in  various  aspects  of  SSA,  including  the  campaign¬ 
modeling  and  analysis  for  which  capability  is  seen  to  have  plummeted, 
and  those  giving  low  marks  to  SSA  processes  and  some  products.  The 
concerns  are  recounted  here  as  received:  During  the  interview  phase, 
I  was  listening  rather  than  assessing  the  correctness  or  fairness  of 
comments. 

A  Weakened  Analytic  Base  for  Valuable  Work 

A  strongly  felt  concern  was  that  because  of  CAPE’s  dissolution  of  cam¬ 
paign-modeling  capability  in  2011,  the  2014  Quadrennial  Defense 
Review  (QDR)  did  not  have  sufficient  analytic  grounding  for  its 
announced  strategy  and  programs.  More  generally,  most  of  the  inter¬ 
viewees  agreed  that  OSD  and  the  Joint  Staff  no  longer  have  much 
internal  ability  to  conduct  detailed  walk-throughs  of  joint  operations 
and  combat-in-planning  scenarios  at  the  campaign  level.4  This  was  said 
to  be  true  with  respect  to  current,  midterm,  and  longer-term  planning. 
The  problem  is  not  just  that  campaign-modeling  is  no  longer  supported 
but  that  the  related  expertise  has  greatly  diminished.  Some  used  the 
phrase  the  baby  was  thrown  out  with  the  bathwater. 

Interviewees  also  said  that  the  military  services  no  longer  have 
the  common  databases  and  baseline  cases  needed  to  ensure  that,  where 
appropriate,  analyses  and  program  development  can  be  based  on  the 
same  joint  data  and  assumptions.  This  means  that  the  different  ser¬ 
vices’  analyses  may  be  unnecessarily  confusing  to  OSD  and  Joint  Staff 
leadership.  Avoiding  that  confusion  was  an  important  objective  in  cre¬ 
ating  SSA  and  the  earlier  analytic  agenda  process.  Also,  a  given  ser¬ 
vice’s  analysis  may  be  less  accurate  than  desired  in  representing  joint 
operations  and  the  contributions  of  its  sister  services.  This  is  troubling 
to  the  individual  service  chiefs,  who  intend  to  be  accurate. 
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Questionable  Value  to  Policymakers 

The  second  class  of  concerns  explained  the  decision  by  CAPE  to  dissolve 
campaign-modeling  capabilities.  To  a  number  of  senior  officials  at  that 
time,  SSA  activity  appeared  to  be  neither  credible  nor  useful,  particu¬ 
larly  when  associated  with  large  and  complex  campaign  models.  The 
more  that  officials  learned  about  the  models,  they  said,  the  less  weight 
they  gave  the  models  in  their  thinking.  Such  models  generate  very  dif¬ 
ferent  outcomes  depending  on  the  many  knob  settings  (assumptions 
about  input  variables),  which  are  too  numerous  and  buried  to  be  easily 
understood.5  Officials  also  expressed  unhappiness  about  responsiveness 
and  agility,  sometimes  mentioning  that  building  scenarios  and  baseline 
analyses  can  take  18  months  and  that  the  analysts  had  been  unable 
to  quickly  answer  “what  if”  questions  when  that  required  changes  of 
important  assumptions.  Briefers  reportedly  sometimes  responded  to 
questions  from  senior  staff  or  policymakers  with,  “Well,  that  is  what 
the  model  said”  (a  cardinal  sin  in  the  minds  of  good  analysts).  In  some 
cases,  senior  officials  seeking  details  during  spot  checks  perceived  some 
of  the  underlying  data  and  assumptions  to  be  noncredible. 

To  others,  the  analyses  did  not  contain  the  information  necessary 
for  making  strategic-level  trade-offs.  Many  officials  in  OSD,  the  Joint 
Staff,  and  service  staffs  criticized  the  narrowness  of  the  analyses,  which 
did  not  address  the  multidimensional  uncertainty  with  which  policy¬ 
makers  are  afflicted  (e.g.,  what  conflicts  might  arise,  in  what  circum¬ 
stances,  and  with  adversaries  using  which  strategies  and  capabilities). 
This  lament  was  consistent  with  the  independent  critiques  since  the 
mid-1990s  that  motivated  the  Secretary  of  Defense  to  refocus  defense 
planning  on  ensuring  broad  future  capabilities  rather  than  capabilities 
only  for  narrow,  special  cases.6 

Several  interviewees  expressed  a  different  concern:  The  data  pro¬ 
vided  to  SSA  by  services  reflect  programs  of  record  and  traditional 
concepts  of  operation  (CONOPS),  which  are  not  forward-looking  or 
creative.  Even  if  the  service  in  question  is  studying  advanced  concepts 
that  it  tentatively  plans  to  adopt,  these  concepts  are  not  reflected  in 
the  SSA  data  (or  the  service  program  objective  memoranda).  Although 
interviewees  understood  the  services’  reluctance  to  assume  success 
of  advanced  concepts,  they  noted  that  national  force  planning  today 
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demands  advanced  concepts  because  advances  in  the  capabilities  of 
potential  adversaries  have  seriously  eroded  the  advantages  previously 
enjoyed  by  the  United  States.7  Further,  using  traditional  concepts  may 
amount  to  inflating  requirements.  As  a  variant  of  the  same  point,  it  was 
said  repeatedly  that  service  inputs  to  SSA  are  based  on  buried  assump¬ 
tions  with  no  transparency.  Other  interviewees  expressed  caution 
about  this — pointing  out  that  the  services  have  reasons  for  conserva¬ 
tism:  Often,  neither  future  nor  current  conflicts  play  out  as  envisioned. 
Also,  changes  of  capability  and  doctrine  often  do  not  develop  as  hoped 
(something  probably  evident  to  all  readers,  given  U.S.  military  history 
since  2001).  Nonetheless,  the  data  provided  to  SSA  are  problematic. 
What  good  are  SSA’s  common  databases  if  senior  OSD  officials  do  not 
trust  them  or  consider  them  appropriate? 

Because  of  such  judgments,  these  officials  did  not  see  the  disband¬ 
ing  of  the  campaign-analysis  activity  as  a  significant  loss.  They  saw  the 
action  as  well  justified,  especially  given  the  austere  budget  environment. 

Summary  of  Concerns 

The  concerns  of  the  two  groups  of  interviewees,  then,  were  seemingly 
contradictory.  Somewhat  more  agreement  existed  about  the  value  of 
SSA’s  collecting  and  sharing  joint  data  than  about  its  model-based  ana¬ 
lytic-baseline  work.  SSA  and  its  predecessor,  the  analytic  agenda,  were 
warmly  credited  with  having  promoted  jointness  by  ensuring  that  offi¬ 
cers  in  the  services  were  better  acquainted  with  their  sibling  services’ 
capabilities,  doctrines,  and  perspectives.  Joint  communication  is  quite 
good  in  comparison  with  years  ago.  That  has  such  benefits  as  allowing 
analysts  to  quickly  support  combatant  commanders  on  current  prob¬ 
lems  or  to  work  effectively  in  special  joint  studies  as  they  arise.  The 
sharing  of  joint  data  facilitates  analysis  across  all  of  DoD. 

That  said,  even  some  of  those  broadly  supporting  SSA  believed 
that  the  detailed  analytic  baselines  prepared  in  earlier  years  had  led  to 
trouble:  The  services  overfocused  on  the  baseline  cases.  Many  inter¬ 
viewees  also  felt  that  the  level  of  detail  in  the  SSA  baselines  had  been 
excessive.  That  detail  went  well  beyond  specifying  objectives  and  broad 
strategy — a  proper  function  of  the  Office  of  the  Under  Secretary  of 
Defense  for  Policy  (OUSDP) — and  instead  specified  scenarios  and 
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CONOPS.  Further,  these  interviewees  believed  that  focusing  analy¬ 
sis  on  verifying  the  executability  of  a  particular  detailed  scenario  and 
CONOPS  discouraged  seeking  the  broader  capability  analysis  needed 
for  planning  under  uncertainty.  This  was  seen  as  a  serious  departure 
from  DoD’s  intended  emphasis  on  capabilities. 

Table  1.1  summarizes  primary  findings  from  the  interviewing 
phase  of  research.  This  report  as  a  whole  aims  to  illuminate  issues  and 
resolve  some  of  the  disagreements  noted  above  and  in  Table  1.1. 


First-Phase  Conclusions  and  the  Structure  of  the  Report 

It  was  evident  from  the  first  phase  of  research  and  subsequent  review 
of  documents  and  relevant  literature  that  the  problems  with  DoD’s 


Table  1.1 

Summary  of  Interview  Information 


Issue 

Views 

SSA's  value  for  promoting  jointness 

Consensus  that  value  was  high 

The  value  of  SSA's  joint  data 

Consensus  that  value  was  high 

The  value  of  SSA's  detailed  analytic 
baselines 

Strong  disagreement  about  value  being 
negative,  low,  or  high 

SSA's  responsiveness,  agility,  and 
understandability 

Consensus  that  problems  exist  and  that 
senior  leaders  complain  about  them; 
disagreements  about  seriousness  and 
inevitability 

SSA's  dealing  with  uncertainties  and 
supporting  capability  planning 

Consensus  that  weaknesses  exist  but 
disagreements  about  seriousness 

Program  creativity  within  the  planning, 
programming,  budgeting,  and 
executing  (PPBE)  process 

Consensus  that  there  is  a  lack  of  creativity 
but  with  disagreements  about 
resolvability 

Logrolling  by  services  and  Joint  Staff 
(insufficient  debate  about  service 
programs  or  discussion  of  emerging 
capabilities  because  participants 
accept  each  other's  inputs) 

Disagreement  between  OSD  and  the 
military  components 

The  purpose  of  SSA 

Disagreements,  except  with  respect  to 
facilitating  sharing  of  some  joint  data 
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capabilities  for  joint  analysis  are  substantial,  that  strong  differences  of 
opinion  exist,  and  that  the  root  problems  run  rather  deep. 

Rather  than  making  on-the-margin  tweaks  to  SSA,  DoD  should 
make  fundamental  revisions  to  the  overall  planning  construct  to  which 
SSA  contributes.  Doing  so  will  lead  naturally  to  changes  in  functions, 
organization,  process,  methods  and  tools,  and  staffing. 

The  remaining  structure  of  the  report  is  as  follows:  Chapter  Two 
goes  back  to  fundamentals  by  asking  what  the  functions  of  joint  analy¬ 
sis  should  be  and  what  kind  of  infrastructure  of  capabilities  is  needed 
to  perform  the  functions.  Chapter  Three  then  subjectively  assesses  cur¬ 
rent  SSA  using  the  structure  from  Chapter  Two.  The  chapter  goes  on 
to  diagnose  problems  and  identify  core  assumptions  and  beliefs  that 
need  to  be  revisited.  It  also  notes  enduring  tensions  that  make  DoD’s 
joint  analysis  difficult.  Chapter  Four  is  more  prescriptive  with  respect 
to  basic  concepts,  organization  and  process,  methods  and  tools,  and 
staffing.  Chapter  Five  provides  brief  recommendations. 


Some  Comments  on  Sources 

The  first  research  phase  of  research  involved  more  than  30  interviews 
with  current  or  past  civilian  and  military  officials,  as  well  as  with  work¬ 
ing  analysts.  The  subsequent  phase  included  selective  review  of  writ¬ 
ten  materials,  primarily  from  1992  onward,  but  with  examples  from 
the  1970s  and  1980s.8  Some  materials  were  formal  U.S.  government 
reports;9  others  were  peer-reviewed  publications  by  the  National  Acad¬ 
emies  of  Sciences  and  Engineering  or  federally  funded  research  and 
development  centers  (FFRDCs);10  still  others  were  briefings  given  at 
conferences  or  elsewhere.11  A  few  of  the  materials  describe  practices 
of  and  lessons  learned  by  allied  defense  ministries,  particularly  in 
the  United  Kingdom,  Canada,  and  Australia.12  It  also  looked  briefly 
at  some  of  the  classic  defense-planning  literature  dating  back  to  the 
1960s.13  Not  surprisingly,  some  of  today’s  issues  were  visible  then. 


CHAPTER  TWO 


Functions  of  Joint  Analysis  and  Attributes  of 
Related  Infrastructure 


This  chapter  establishes  a  framework  for  assessing  capabilities  for  joint 
analysis.  The  first  section  discusses  functions  to  be  accomplished;  the 
second  section  discusses  the  attributes  of  infrastructure  needed,  par¬ 
ticularly  the  part  of  infrastructure  to  which  SSA  relates. 


Functions 

Strategic  planning  is  an  organization’s  process  of  considering  objec¬ 
tives,  formulating  strategy,  and  making  decisions  about  how  to  allo¬ 
cate  resources.  It  might  also  include  control  mechanisms  for  monitor¬ 
ing,  feedback,  and  adaptation.  Strategic  planning  and  strategic  analysis 
occur  at  multiple  levels  and  in  a  distributed  manner.  The  Secretary  of 
Defense  does  strategic  planning  but  so  do  others  within  DoD,  such  as 
the  service  chiefs,  the  Chairman  of  the  Joint  Chiefs  of  Staff,  and  Under 
Secretaries  of  Defense. 

Analysis  could  serve  a  number  of  generic  functions,  such  as 
(1)  directly  aiding  decisionmakers,  (2)  broadly  supporting  decision¬ 
makers  by  providing  expertise  and  undergirding  higher-level  decision- 
aiding,  (3)  supporting  execution  of  decisions,  and  (4)  ensuring  that 
operation  planning  (e.g.,  war  planning)  adheres  to  strategy  and  policy. 
These  functions  are  discussed  in  the  following  subsections.1 


7 


8  Capabilities  for  Joint  Analysis  in  the  Department  of  Defense 


Decision-Aiding 

In  this  context,  the  word  aid  conveys  a  sense  of  direct  and  personal 
assistance,  rather  than  just  broad  indirect  support.  Ideally,  decision- 
aiding  for  high  officials  should2 

1.  frame  issues  in  terms  of  objectives,  criteria,  and  goals 

2.  develop  and  present  options 

3.  comprehensibly  compare  the  options,  with  pros  and  cons  or  by 
using  parametrics  for  trade-offs  and  cost-effectiveness  analysis 

4.  provide  the  real-time  ability  to  “zoom”  or  “drill  down”  into 
detail  as  necessary  to  explain  particular  high-level  results  or  the 
effect  of  assumptions3 

5.  discuss,  across  categories,  how  to  allocate  resources  so  as  to  bal¬ 
ance  portfolios  (e.g.,  to  decide  on  mixes  of  capabilities  and  mixes 
of  ways  to  achieve  capabilities,  so  as  to  appropriately  address  all 
objectives  within  a  budget). 

The  first  four  are  necessary  to  address  broad  strategic  issues  and  for  a 
myriad  of  component  issues.  Tensions  and  conflicts  occur  across  these 
items,  so  the  fifth  one  is  essential.  Decision-aiding  is  needed  by  the  sec¬ 
retary,  chairman,  under  secretaries,  and  service  chiefs. 

Such  aiding  could  be  accomplished  in  personal  discussions,  a 
short  memorandum  or  briefing,  or  a  sizable  study.  In  any  case,  the 
material  should  summarize  from  a  much  larger  body  of  knowledge  and 
understanding  of  disputes.  Such  high-level  analysis  is,  then,  in  part  a 
matter  of  asking  the  right  questions,  framing  the  issues,  identifying  cri¬ 
teria  and  goals,  and  providing  insights — which  is  distinct  from  finding 
the  optimum  solution  to  a  well-posed  math  problem.  All  such  aiding 
must  be  comprehensible,  credible,  and  suitable  for  iteration  with  deci¬ 
sionmakers.  Such  decision-aiding  is  usually  the  responsibility  of  elite 
analytic  groups  working  personally  for  the  decisionmaker.4  It  is  not  a 
natural  function  for  committees. 

Broad  Underlying  Support  of  Strategic  Decisionmaking 

Broad  support  for  decisionmaking  includes  ensuring  that  knowledge  is 
available  when  needed  on  diverse  subjects.  Broad  support  includes  pro- 
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viding  a  deeper  body  of  knowledge  and  information.  After  all,  high- 
level  decision-aiding  depends  on  many  simplifications  and  judgments. 
These  should  be  informed  by  broader  and  deeper  knowledge,  often 
based  on  research,  analysis,  and  empirical  data.  Table  2.1  illustrates 
the  kinds  of  DoD  issues  on  which  such  deeper  knowledge  has  been 
needed  in  the  past.  Each  of  the  questions  led  to  its  own  joint  study 
drawing  on  SSA  information.  Ideally,  the  products  can  flow  upward, 
as  suggested  in  Figure  2.1.  The  figure  anticipates  that  decisionmakers 
will  use  their  own  strategic  analysts  to  produce  the  needed  products. 
Ideally,  however,  these  products  will  be  informed  by  deeper  analysis. 
As  an  example,  while  preparing  an  analysis  on  strategic  mobility  for 
decisionmakers,  it  is  of  little  use  to  know  all  the  data  that  went  into  a 
detailed  study  showing  that  a  particular  deployment  could  be  accom¬ 
plished  with  the  projected  level  of  airlift.  Rather,  the  analysis  should 
show  parametrics  from  such  deeper  analysis,  parametrics  indicating 
what  kinds  of  deployments  could  and  could  not  be  supported.  The 
parametrics  should  be  consistent  with  (although  abstracted  from) 
more-detailed  work. 


Table  2.1 

Illustrative  Questions  in  Need  of  Deep  Analysis 


Question 


What  capabilities  does  the  United  States  need  to  find  and  neutralize  weapons  of 
mass  destruction? 

What  capabilities  does  the  United  States  need  to  execute  its  strategy  for  the  war  on 
terror? 

What  is  the  best  mix  of  tactical  air  forces? 

What  is  the  best  composition  (number,  mix,  and  characteristics)  of  the  future  tanker 
fleet? 

What  capabilities  does  DoD  need  to  support  civil  authorities  when  responding  to  a 
domestic  disaster? 

Are  U.S.  projected  future  forces  sufficient  to  meet  the  defense  strategy? 


SOURCE:  Stevens,  2008. 
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Figure  2.1 

Relationship  Between  Broad  Support  and  Decision-Aiding 


Report  to,  discuss  with  policymakers 

A 

k 

•  Framework,  story,  and  optional  "zooms" 
for  explanations,  trades,  and  what-ifs? 

•  Expertise  permitting  immediate  response 
to  other  queries 

•  Fast  reach-back  capability 

Prepare  at-time  decision-aiding 

SOURCE:  Davis,  2014,  Figure  3.1. 

RAND  RR1469-2.  1 


Where  such  deeper  knowledge  is  needed,  supporting  analysis 
should  generally 

1.  reflect  a  full  understanding  of  the  challenges,  conceiving  the 
system  broadly 

2.  consider  broad  and  creative  options  and  alternative  ways  to 
frame  issues 

3.  provide  clear  and  rigorous  definitions,  distinctions,  assump¬ 
tions,  metrics,  and  goals 

4.  provide  multiresolution  analysis  of  uncertainties  and  disagree¬ 
ments  (e.g.,  parametric  trade-offs  at  different  levels  of  detail,  as 
needed) 

5.  arrange  for  research  to  better  understand  the  phenomena 
involved  in  the  challenges  and  obtain  data 
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6.  winnow  down  by  suggesting  simplified  specialized  frames,  mea¬ 
sure  hierarchies,  and  best-of-breed  options  for  each  option  type 
to  be  sent  up  the  line  and  perhaps  used  in  high-level  decision- 
aiding 

7.  for  all  of  these  items,  focus  on  how  to  achieve  flexibility,  adap¬ 
tiveness,  and  robustness  (FARness) — rather  than  solutions  fine- 
tuned  to  precise  but  fragile  assumptions  about  the  future.5 

The  seventh  item  has  not  been  recognized  adequately  in  traditional 
supporting  analysis.  This  item  defines  a  new  demand  that  policymak¬ 
ers  should  place  on  analysis  and  a  new  professional  responsibility  for 
analysts  that  goes  well  beyond  identifying  assumptions,  as  good  ana¬ 
lysts  already  do.6 

Supporting  Execution 

A  sometimes-neglected  element  of  analysis  for  strategic  planning  is 
supporting  the  execution  of  decisions.  This  includes  the  following: 

1.  Interpret  and  tighten  the  decision  and  then  propagate  more- 
detailed  guidance,  including  measures,  metrics,  and  goals.7 
Without  this,  guidance  may  be  difficult  for  components  to 
follow  and  too  vague  to  force  desired  changes.8 

2.  Monitor  performance  and  outcome  from  related  initiatives  and 
supporting  lower-level  adaptations:  Are  programs  funded  and 
executed  as  intended  (inputs);  are  they  proceeding  as  intended; 
are  their  outputs  as  intended? 

3.  Flag  variations  from  the  plan  that  imply  the  need  for  review  and 
new  top-level  decisions.  Examples  might  involve  failure  assump¬ 
tions  critical  to  the  original  decision:  changes  in  the  world,  tech¬ 
nical  failures,  intolerable  cost  growth,  or  discovery  of  fatal  flaws 
in  the  CONOPS. 

Getting  the  metrics  right  is  difficult.  Organizations  often  come 
up  with  metrics  that  seem  appropriate  superficially  but  are  actually 
useless  or  worse  because  they  create  perverse  incentives.  Sound  metrics 
need  to  be  derived  analytically  from  an  understanding  of  operations.  In 
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many  cases,  these  metrics  turn  out  to  be  natural  parameters  of  a  good 
model  of  the  operation  and  also  to  fall  into  natural  levels  of  hierarchi¬ 
cal  detail.9  Using  higher-level  parameters  that  are  indifferent  to  details 
of  how  an  operation  is  accomplished  can  avoid  some  of  the  pernicious 
effects  of  metrics  that  presuppose  such  details. 

Ensuring  Adherence  to  Strategy  and  Policy  in  Operational  Planning 

Another  function  of  joint  analysis  is  to  ensure  that  the  intentions  of 
DoD’s  strategy  and  policy  are  properly  reflected  in  operational  plan¬ 
ning,  such  as  war  planning  by  combatant  commanders,  and  in  the 
guidance  that  they  receive  through  the  Joint  Staff  (e.g.,  the  Joint  Strate¬ 
gic  Capabilities  Plan).  Most  aspects  of  this  are  classified,  but  the  inten¬ 
tions  are  indicated  by  definitions  and  legislation.10  This  report  does  not 
address  such  matters. 

Functions  of  SSA 

Against  this  background  of  generic  functions,  the  SSA  activity  was 
intended  primarily  to  accomplish  two  purposes  as  specified  by  a  DoD 
directive  (DoD  Directive  8260.05,  2011):  (1)  Support  deliberations  by 
DoD  senior  leadership  on  strategy  and  PPBE  process  matters,  includ¬ 
ing  force-sizing,  force-shaping,  and  force-capability  development,  and 
(2)  provide  a  starting  point  for  studies  that  support  development  and 
implementation  of  defense  strategy  and  policy  and  support  DoD’s 
PPBE  process. 

The  five  functions  of  analysis  are  fulfilled  in  different  ways.  Deci¬ 
sion-aiding  is  primarily  the  responsibility  of  staffs  working  directly  for 
the  decisionmaker.  SSA’s  role  has  been  largely  to  provide  broad  under¬ 
lying  support  in  the  form  of  expertise,  joint  data,  and  joint  analysis 
baselines  (the  shaded  area  in  Table  2.2).  SSA  has  not  played  much  of 
a  role  in  supporting  program  execution;  for  the  Secretary  of  Defense, 
that  function  is  largely  served  by  CAPE,  assisted  by  the  Joint  Staff  and 
by  service  staffs.  SSA  is  also  only  playing  a  limited  role  in  ensuring 
adherence  of  war  plans  to  strategy,  although  it  could  do  more. 

A  cooperative,  consensus-seeking  activity  such  as  SSA  cannot  rea¬ 
sonably  be  very  useful  in  directly  aiding  decisionmaking  by  the  Sec¬ 
retary  of  Defense  (the  activity  can  be  more  useful  for  aiding  service 
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Table  2.2 

The  Support  for  Strategic  Analysis  Role  in  Joint  Analysis 


Decision- 

Aiding 

Broad  Support 

Ensuring 

Support  of  Adherence  to 
Execution  Strategy 

Undergirding 
Expertise  Higher-Level 
Analysis 

Current  role  • 

of  SSA 

Possible  • 

future  role  of 
an  SSA-like 
activity 

••••  •• 

••••  •••• 

•  • 

NOTE:  More  bullets  in  a  cell  indicates  a  greater  emphasis  on  the  column's  function. 


chiefs)  (A.  Barber,  2014a,  2014b),  but  the  activity  can  provide  broad 
support  in  the  form  of  expertise  and  the  undergirding  of  higher-level 
analysis.  A  future  SSA-like  activity  could  do  much  more  in  this  respect, 
with  its  results  flowing  coherently  into  modern,  uncertainty-sensitive, 
higher-level  analysis  for  decision-aiding  (Figure  2.1).  The  activity  could 
also  play  more  of  a  role  in  supporting  execution  and  ensuring  adher¬ 
ence  to  strategy.  Any  such  changes,  if  desired,  would  require  significant 
changes  in  expectations,  organization,  tasking,  process,  methods,  and 
staffing. 


Attributes  of  Infrastructure 

After  discussing  functions  of  joint  analysis,  it  is  useful  to  ask  what  is 
needed  to  have  a  good  infrastructure  for  the  broad  range  of  analytic 
activities.  In  this  view,  the  outputs  of  the  infrastructure  are  the  qual¬ 
ity  and  responsiveness  of  the  functions.  But  what  attributes  constitute 
a  good  infrastructure,  one  that  can  perform  these  functions?  And,  in 
particular,  what  should  be  the  attributes  of  SSA  when  viewed  as  infra¬ 
structure?  A  reasonable  set  of  attributes  to  consider  is 

1.  diversity  of  suppliers 

2.  healthiness  of  relationships  among  suppliers  and  with  clients 
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3.  sufficiency  of  human  and  intellectual  capital  of  the  suppliers 
(including  methods) 

4.  adequate  processes  to  manage  and  sustain  the  infrastructure. 

For  DoD,  the  infrastructure  for  joint  analysis  must  include  partici¬ 
pants  from  all  the  relevant  DoD  components,  with  participants  under¬ 
standing  their  joint  contexts  and  narrower  ones,  in  addition  to  being 
able  to  coordinate  and  even  collaborate  well. 

The  components  of  DoD  are  competing  for  resources,  which 
strongly  affects  individual  incentives  with  DoD.11  A  good  infrastruc¬ 
ture  for  DoD  should  make  it  possible  to  manage  such  competitions 
fairly  (ensuring  a  level  playing  field  when  comparing  service  programs), 
wisely,  and  effectively  at  reasonable  cost.  That  implies  requirements 
for  openness,  sharing,  and  common  basic  data.  In  addition,  it  requires 
management  attention  and  resources  to  keep  the  infrastructure  diverse, 
vibrant,  learning,  and  keeping  up  with  and  even  anticipating  new  chal¬ 
lenges.  The  next  chapter  evaluates  SSA  by  these  criteria  and  those  given 
earlier  in  this  chapter. 


CHAPTER  THREE 

Evaluation  and  Diagnosis 


Base  Evaluation 

Evaluation  of  Functions 

Congress  asked  for  an  assessment  of  DoD’s  current  capabilities  for 
joint  analysis  (see  Appendix  A).  Resources  did  not  permit  an  exhaus¬ 
tive  study.  What  follows  are  my  subjective,  though  informed,  assess¬ 
ments  using  the  structure  laid  out  in  Chapter  Two,  the  considerable 
interview  information  gathered  during  the  study,  and  previous  experi¬ 
ence  and  research  (see  Davis,  2014).  The  assessments  appear  approxi¬ 
mately  correct,  and  some  scores  are  given  as  ranges.  Significantly,  a 
given  reader  might  disagree  with  the  precise  subjective  scores  assigned 
while  still  being  convinced  of  the  shortcomings  identified  in  the  sub¬ 
sequent  discussion. 

Table  3.1  defines  the  qualitative,  subjective  assessments  on  a  1-9 
scale,  with  1  being  very  poor  and  9  being  very  high. 

This  scaling  is  probably  intuitive  to  readers  who  have  been  con¬ 
sumers  of  analysis.  Some  analysis  lacks  credibility  or  is  irrelevant  to 
decisions,  or  both  (score  of  1).  Perhaps  the  problem  is  low-quality  ana¬ 
lysts,  but  more  likely  it  is  because  the  issues  are  not  well  understood.  A 
good  deal  of  analysis  merits  a  medium  score  of  5.  Here  the  analysis  is 
credible  as  far  as  it  goes,  relevant,  and  responsive.  It  also  provides  some 
significant  insights.  It  is  limited,  however,  by  narrowness  and  a  some¬ 
what  pedestrian  flavor — answering  specific  questions  but  not  gener¬ 
alizing  beyond  the  specifics.  Also,  this  analysis  might  not  boil  issues 
down  or  cleanly  frame  issues.  At  the  high  end  (a  score  of  9),  analysis  is 
not  only  very  good,  relevant,  and  responsive  but  goes  beyond  that  to 
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Table  3.1  Definitions  of  Scale  Values  in  Subjective  Assessments 


Score  Meaning 


1  Poor,  unresponsive  information;  little  insight;  little  expertise;  no  useful 

abstraction  or  framing 

3  A  mix  of  1  and  5  (i.e.,  examples  of  each) 

5  Good,  semiresponsive  information;  somewhat  narrow  expertise;  some 

insight;  not  much  uncertainty  analysis  or  trade-offs,  imagination, 
anticipation,  framing,  or  useful  abstraction 

7  A  mix  of  5  and  9  or  consistent  in-between  performance 

9  Very  good,  highly  relevant  information;  broad  expertise;  imagination, 

uncertainty  analysis,  anticipation,  framing,  and  useful  abstraction  with 
supporting  analysis 


sharpen  and  expand  the  questions,  anticipate  larger  issues,  and  provide 
more-general  insights  (e.g.,  indicating  under  what  circumstances  results 
would  and  would  not  be  favorable).  Finally,  it  suggests  simplifications. 

All  of  this  begs  the  question  of  whether  we  are  measuring  analysis 
relative  to  what  was  asked  or  relative  to  what  might  have  been  asked. 
Perhaps  policymakers  should  pose  different  and  more-thoughtful  chal¬ 
lenges  for  SSA.  With  that  in  mind,  Table  3.2  provides  this  report’s 
assessments  for  the  current  standard  (i.e.,  for  the  tasking  as  it  has 
existed  and  been  interpreted)  and  for  an  elevated  standard.  The  later 
assessments  are  based  on  my  subjective  sense  of  how  well  the  2011  SSA 
could  have  done  before  the  loss  of  its  campaign-modeling  had  ques¬ 
tions  been  more  demanding. 

Decision-Aiding 

The  second  column  of  Table  3.2  shows  the  assessment  of  SSA  for  the 
decision-aiding  function.  This  has  not  been  a  core  SSA  function,  so  it 
is  not  surprising  that  SSA  has  not  contributed  significantly  to  directly 
aiding  the  secretary  or  under  secretaries.  However,  those  interviewed 
noted  with  examples  that  service  chiefs  have  sometimes  been  directly 
aided.  Thus,  even  if  the  overall  assessment  is  2,  the  range  is  perhaps 
from  1  to  5.  The  score  might  have  been  somewhat  better  had  different 
questions  been  asked  (the  elevated-standard  row). 


Table  3.2 

Top-Level  Subjective  Evaluation  of  Current  Support  for  Strategic  Analysis  as  a  Function  of  Standard  Used 


Score  for  Broadly  Supporting  High-Level 
Decisionmaking 
(current  role  of  SSA) 


Undergird  High-Level 
Expertise  and  Advice  Analysis 

Score  for  Aiding  _  Score  for  Supporting 

High-Level  Score  for  Adherence  to 


Decisionmaking 

2011 

2016 

2011 

2016 

Supporting  Execution 

Strategy  and  Policy 

Current 

standard 

2 

(1-5) 

5 

(4-7) 

4 

(3-5) 

3 

(1-5) 

2 

(1-3) 

N/A 

1-3 

Elevated 

standard 

3 

(1-6) 

6 

4 

5 

3 

Selective 
possibilities: 
definitions,  metrics, 
goals,  and  risks 

Some  role  possible 

Reason 
for  score 

Mismatch  with 
Secretary  of 
Defense  needs 
(better  match 
with  needs  of 
chiefs) 

Problems 
with  level 
of  analysis, 
staffing, 
and  mind¬ 
sets 

Problems 
with  level 
of  analysis, 
staffing, 
mind-sets, 
and  reduced 
capabilities 
since  2011 

Problems 
with  level 
of  analysis, 
staffing, 
and  mind¬ 
sets 

Problems 
with  level 
of  analysis, 
staffing, 
mind¬ 
sets,  and 
reduced 
capabilities 

Problems  with  level 
of  analysis,  staffing, 
mind-sets,  and 
reduced  capabilities 

Levels;  methods; 
and  problems  with 
level  of  analysis, 
staffing,  mind¬ 
sets,  and  reduced 
capabilities 

NOTES:  The  scale  is  1  (very  poor)  to  9  (excellent).  Ranges  are  in  parentheses.  N/A  =  not  applicable. 
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Supporting  the  Execution  and  Review  of  Current  War  Plans 

Skipping  temporarily  to  the  last  two  columns  of  Table  3.2,  the  assess¬ 
ment  is  that  SSA  has  not  been  expected  to  support  the  execution  of 
the  defense  program.  As  seen  in  the  elevated-standard  row,  it  would 
seem  logical  that  SSA  could  have  done  more,  primarily  to  sharpen  defi¬ 
nitions,  suggest  metrics,  provide  analysis  that  suggests  goals  for  the 
metrics,  and  characterize  execution  risks.  That  would  be  very  difficult 
now  because  of  the  reduced  campaign-modeling  capability.1  As  for  sup¬ 
porting  OSD’s  review  of  current  war  plans  (last  column,  on  support¬ 
ing  adherence  to  strategy  and  policy),  the  Deputy  Assistant  Secretary 
of  Defense  for  Plans  called  on  SSA-related  expertise  to  some  extent 
through  2011.  With  the  loss  of  SSA’s  campaign  analysis,  however,  that 
expertise  is  no  longer  available. 

Expertise  and  Undergirding  Higher-Level  Analysis 

The  shaded  columns  in  the  middle  of  Table  3.2  show  assessments  for 
the  core  functions  of  SSA.  These  are  shown  separately  for  the  period 
through  2011,  when  SSA  had  extensive  campaign-modeling,  and  the 
subsequent  (current)  period,  when  SSA  has  not  had  such  modeling. 
The  assessment  is  that,  through  2011,  SSA  provided  significant  exper¬ 
tise  (those  doing  campaign-modeling  were  professional  and  respected).2 
Nonetheless,  SSA  was  judged  as  falling  short  in  terms  of  its  work  under¬ 
girding  the  more-important  aspects  of  high-level  analysis.  Those  more 
directly  in  the  joint  decision-aiding  business  did  not  see  much  benefit 
to  SSA’s  efforts  and  felt  that  the  analytic  baselines  were  getting  in  the 
way  by  underplaying  uncertainty  and  the  search  for  fulsome  capabili¬ 
ties.  Also,  the  efforts  overfocused  on  the  campaign  level,  rather  than 
the  mission  level  where  many  problems  reside.  Some  saw  SSA  as  an 
unproductive  overhead  activity. 

Table  3.3  explains  the  evaluations  of  the  undergirding-strategic- 
analysis  function  using  criteria  from  Chapter  Two.  SSA  has  done  some 
framing,  has  arguably  provided  some  options  (usually  as  provided  by 
the  services),  and  had  done  at  least  a  bit  (score  of  3)  in  comparing 
options.  It  has  not,  however,  done  much  uncertainty  analysis  or  pro¬ 
vided  layered  explanations.  Because  point  cases  are  of  little  interest, 
the  score  assigned  is  3.  SSA  could  have  been  asked  to  do  more  (the  ele- 
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Table  3.3 

Factors  in  the  Assessment  of  the  Support  for  Strategic  Analysis  Function 
of  Undergirding  Fligh-Level  Analysis 


Score  on 
Framing 

Score  on 
Providing 
Options 

Score  on 

Comparing  Options 
Comprehensibly 
and  Credibly, 
with  Uncertainty 
Analysis 
and  Layered 
Explanations  (zoom) 

Score  on 
Allocating 
Resources 

Score  on 
Undergirding 
High-Level 
Analysis  (2011) 

Current 

standard 

5  (3-7) 

4 

3 

N/A 

3 

Elevated 

standard 

5  (3-7) 

5 

5 

N/A 

5 

NOTES:  The  scale  is  1  (very  poor)  to  9  (excellent).  Ranges  are  in  parentheses.  N/A  = 
not  applicable. 


vated-standard  row),  and  the  results  would  probably  have  been  some¬ 
what  better.  SSA  probably  could  not  reasonably  have  been  asked  to 
help  in  allocating  resources.  SSA  depends  on  cross-component  cooper¬ 
ation,  sharing,  and  sometimes  on  consensus.  Such  an  activity  is  seldom 
good  at  resource-allocation  analysis,  which  is  why  organizations  such 
as  CAPE  exist.  Overall,  then,  SSA  did  not  fare  well  in  the  assessment 
(see  the  last  column  of  Table  3.3). 

Evaluation  of  SSA  Infrastructure 

Table  3.4  is  the  assessment  of  SSA  as  infrastructure.  The  shaded  col¬ 
umns  use  the  attributes  described  in  Chapter  Two.  Overall,  the  evalu¬ 
ation  is  moderate  (5)  with  a  sizable  range  (4-7). 

Senior  leaders  see  relatively  little  of  what  goes  on  at  the  SSA  level. 
They  may  not  easily  appreciate  such  infrastructure.  I  believe,  however, 
based  on  past  experience  and  the  testimony  of  those  interviewed  for 
this  study,  that  the  SSA  infrastructure  has  been  very  important  in 
DoD’s  nurturing  of  jointness.  Officers  and  analysts  who  would  oth¬ 
erwise  be  stovepiped  in  their  services  or  OSD  offices  mingle  and  work 
together  routinely  and  well — learning  about  each  other’s  capabilities, 
doctrine,  special  features,  and  idiosyncrasies  (including  those  of  lan- 
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Table  3.4 

Evaluation  of  Support  for  Strategic  Analysis  as  Infrastructure 


Products  of  SSA 
Infrastructure 

Attributes  of  Infrastructure 

Assessment 

Score  on 

Score  on  Responsive- 
Quality  ness 

Score  on 
Diversity  of 
Suppliers 

Score  on 
Relationships 
Among 
Suppliers 
and  Clients 

Score  on 
Human 
and 

Intellectual 

Capital 

Score  on 
Infra¬ 
structure 
Oversight 

Score  on 
Evaluation 
of  Infra¬ 
structure 
Attributes 

5  (3-7)  3 

3 

7  (4-8) 

5  (3-7) 

6  (3-7) 

5  (4-7) 

Some  Poor3 

high- 

quality 

products; 

narrow 

Weak  on 
technology 
push  and 
advanced 
concepts 

Mostly 

good  given 

enduring 

tensions; 

inadequate 

confrontation 

Good  but 
narrow 

Limited  by 
authorities, 
complexity, 
consensus, 

resources; 

includes 

some 

research 

a  SSA-associated  analysts  have  often  been  responsive  to  their  immediate  leaders  in 
CAPE,  OUSDP,  the  Joint  Staff,  and  the  services.  The  analysts'  responsiveness  has  often 
benefited  from  knowledge  and  data  gained  as  part  of  the  SSA  activity. 


guage).  These  officers  and  analysts  learn  the  trade  of  joint  analysis  by 
using  common  methods  and  models.  This  is  akin  to  having  common 
textbooks.  This  pays  off  when  Joint  Staff  analysts  are  called  on  to  help 
combatant  commander  staffs  in  theaters  of  war  or  when  DoD  has  an 
important  joint  study.  SSA,  however,  has  had  weak  connections  with 
the  worlds  of  technology  and  innovation  (e.g.,  the  focus  of  the  recently 
created  Strategic  Capabilities  Office,  which  reports  to  the  Deputy  Sec¬ 
retary  of  Defense). 


Summary  Evaluation  and  Diagnosis 

In  summary,  this  report’s  assessment  of  SSA  is  not  very  positive.  Fur¬ 
ther,  SSA’s  capabilities  and  value  have  dropped  significantly  because  of 
the  loss  of  SSA’s  capability  for  campaign-modeling.  In  my  view,  how¬ 
ever  (consistent  with  many  of  the  inputs  received), 

•  restoring  past  campaign-modeling  capability  would  not  resolve 
the  largest  problems 
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•  major  changes  are  needed  and  feasible,  as  discussed  in  Chapter 

Four 

•  the  basic  problem  is  that  current  SSA  has  severe  limitations  of 

methods,  tools,  staffing,  and  mind-set. 

This  assessment  is  not  a  criticism  of  SSA’s  efforts.  SSA  had,  by  2010- 
2011,  become  effective  at  what  it  had  been  asked  to  do  and  was  on 
a  path  toward  doing  even  better.3  SSA’s  history  includes  numerous 
bright  spots,  including  efforts  to  address  difficult  problems  relating  to 
counterterrorism,  counterinsurgency,  and  intervention.4  SSA  has  much 
to  be  proud  of.  That  said,  for  a  revitalized  SSA  to  achieve  the  elevated 
standards  that  it  should  be  given  to  better  serve  policymakers,  major 
changes  will  be  needed  in  concepts,  process,  organization,  and  staffing. 
That  will  require  some  rethinking  by  senior  leaders  as  they  consider  a 
redesign. 

Comparison  with  an  Earlier  Assessment  of  DoD  Analytic  Culture 

Many  issues  are  long-standing.  Table  3.5  is  taken  from  DoD  modeling 
and  simulation  (M&S)  conducted  in  1996  (Defense  Science  Board, 
1996a).  Some  of  the  shortcomings  that  the  Defense  Science  Board 
noted  of  DoD  analysis  culture  have  been  mitigated  since  then.  DoD 
now  considers  many  more  scenario  types,5  and  it  has  more  joint  data 
to  support  analysis.  It  has  also  taken  a  closer  look  at  such  complex 
phenomena  as  counterterrorism  and  counterinsurgency.  Other  short¬ 
comings,  however,  are  the  same  as  they  were  20  years  ago:  complex 
bureaucratic  processes,  suppression  of  deeper  forms  of  uncertainty,  and 
“mechanical”  models  (i.e.,  models  that  are  longer  on  computation  than 
on  providing  insight). 

The  Defense  Science  Board  study  also  suggested  contributions  to 
infrastructure,  as  indicated  in  Table  3.6.  The  study  suggested  a  ded¬ 
icated  research  organization  established  by  the  Director  for  Defense 
Research  and  Engineering,  what  is  now  roughly  the  Assistant  Secre¬ 
tary  of  Defense  for  Research  and  Engineering,  to  study  information 
dominance,  long-range  precision  strike,  and  operational  concepts. 
This  research  was  to  be  tied  to  the  warfighting  community  and  would 
be  expected  to  solve  new  problems.  The  Defense  Science  Board  also 
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Table  3.5 

Changes  Needed  in  the  DoD  Analysis  Culture  (from  Defense  Science  Board 
Study) 


What  Is 

What  Should  Be 

Closed  processes 

Open  processes 

Bureaucratic  review 

Peer  review 

Accredited  analysis 

Competitive  analysis 

Model  orientation 

Subject-matter  orientation 

Mechanical 

Meaningful 

Data-poor 

Data-rich 

Rigid  approvals 

Learning  and  adaptation 

Stable  algorithms 

Unstable  phenomena 

Suppressed  uncertainty 

Illuminated  uncertainty 

Suppressed  risk 

Illuminated  risk 

Narrow,  Cold  War  style 

Oriented  to  present  and  future 

Few  accredited  scenarios 

Many  political-military  scenarios 

Point  assumptions 

Exploratory  analysis 

SOURCE:  Adapted  from  Defense  Science  Board,  1996a  (a  report  chaired  by  retired  Air 
Force  Major  General  Jasper  Welch). 


envisioned  a  holistic  view  of  M&S  within  analysis,  as  indicated  in 
Figure  3.1. 

The  Defense  Science  Board’s  suggestions  pertained  to  DoD  joint 
analysis  broadly,  not  just  to  what  became  the  work  of  the  later  analytic 
agenda  and  SSA.  SSA  has  been  tied  more  to  PPBE  than  to  acquisitions. 
Some  of  the  Defense  Science  Board’s  recommendations  have  been  pur¬ 
sued  under  sponsorship  of  USD(AT&L).  USD(AT&L),  for  example, 
sponsored  the  creation  of  the  Joint  Advanced  Warfighting  Program  at 
the  Institute  for  Defense  Analyses. 

In  some  respects,  it  has  been  desirable  for  such  work  to  be  sepa¬ 
rated  from  that  of  PPBE  because  of  the  cultural  differences  between 
communities.6  One  consequence,  however,  has  been  that  SSA  has  suf- 
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Table  3.6 

A  Research  Approach  Suggested  in  1996 


Mission  Statement 


Examples 


Identify  analytical  issues  needing 
improvement  derived  from  both 
customer  concerns  and  research 

Collect  and  analyze  empirical  data  from 
all  sources,  both  existing  and  program¬ 
generated 

Collect  and  analyze  results  from  all  types 
of  M&S,  both  existing  and  program¬ 
generated,  at  all  relevant  levels  of 
resolution 

Create  and  maintain  an  overall 
intellectual  framework  by  engaging 
customers,  actively  guiding  the  research 
program,  and  stimulating  both  peer 
review  and  open  debate 

Serve  the  customer  community  by 
providing  expert  advice  and  advisors, 
making  available  new  and  better 
analytical  modules  for  widespread  use, 
and  evaluating  analysis  at  request  of 
customers 


Countermeasures,  realistic  effectiveness 
calculations,  behavioral  assumptions  at 
both  small-unit  and  commander  level 

History,  structured  interviews, 
instrumental  training  exercises,  virtual 
exercises,  field  tests 

Encourage  integrated  hierarchical  families 
of  models,  including  selectable  resolution; 
exploit  data  from  all  levels  of  resolution 


Provide  strong  problem  definition  and 
analytic  plans  to  those  conducting  virtual 
experiments.  Create  subject-area  forums 
for  in-depth  exchange  and  peer  review 


Provide  red  teaming,  definitive 
effectiveness  calculations,  and  analytically 
sound  modules  (while  recognizing 
uncertainty) 


NOTE:  Retyped  and  reformatted,  but  otherwise  identical  to  original  (Defense 
Science  Board,  1996a). 


fered  from  inadequate  technology  push  and  innovation  push.'  Also, 
DoD  as  a  whole  has  nothing  similar  to  the  program  suggested  by 
Figure  3.1.  Some  related  suggestions  appear  in  Chapter  Four. 

Comparison  with  Suggestions  from  a  National  Academies  Study 

As  another  data  point  to  demonstrate  that  the  problems  noted  in 
Table  3.5  and  earlier  in  the  report  are  long-standing,  Table  3.7  men¬ 
tions  selected  recommendations  of  a  National  Academies  of  Sciences, 
Engineering,  and  Medicine  study  from  2006.  The  study  was  supported 
by  the  Defense  Modeling  and  Simulation  Office8  through  an  initia¬ 
tive  of  CAPE.  The  panel  referred  to  modeling,  simulation,  and  analy¬ 
sis  (MS&A)  to  emphasize  that  DoD’s  focus  on  M&S  was  somewhat 
wrong-headed  because  it  omitted  the  crucial  element  of  high-quality 
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Figure  3.1 

A  Holistic  Approach  Suggested  in  1996 


SOURCE:  Adapted  from  Defense  Science  Board,  1996a. 

NOTES:  Research  includes  empirical  work  and  model-based  work  that  vary  in  levels  of 
resolution,  types,  and  degrees  and  human  and  analytical  exploration  of  uncertainty 
and  sensitivity.  The  intent  is  to  use  or  generate,  and  then  integrate,  all  knowledge 
of  a  phenomenon. 

RAND  RR1 469-3.1 


Table  3.7 

Selected  Recommendations  from  2006 


Some  Synopsized  Recommendations 


Modern  technology  and  CONOPS :  Ensure  that  the  basic  architecture  of  MS&A 
systems  reflects  modern  concepts  of  network-centric  warfare. 

Broad  approach  to  analysis  and  methods'.  Invest  in  a  range  of  methods,  including 
diverse  models,  games,  field  experiments,  and  other  ways  to  obtain  information. 

Social-science  methods :  Devote  significant  research  to  social  behavioral  networks 
and  multi-agent  systems  because  both  are  promising  approaches  to  the  difficult 
modeling  challenges. 

Uncertainty:  Seek  better  methods  to  characterize,  quantify,  and  manage  the 
uncertainty  inherent  in  all  aspects  of  MS&A — including  inputs,  modeling 
assumptions,  parameters,  and  options. 

Interface  with  decisionmakers:  Strive  to  understand  cognitive  styles  of 
decisionmakers  and  their  interaction  with  different  forms  of  MS&A. 

DoD  should  seek  better  methods  to  characterize  interface. 

Broad  research  on  military  science:  Identify  (or  create)  and  charge  an  organization 
with  responsibility  for  developing  and  supporting  research  and  development  to 
improve  and  update  base  of  military  science  for  combat  and  noncombat  modeling. 


SOURCE:  Adapted  from  National  Research  Council,  2006. 
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analysis — something  that  does  not  follow  easily  from  M&S  by  itself 
and  is  sometimes  frustrated  by  the  nature  of  M&S.  By  and  large,  the 
recommendations  have  not  as  yet  been  heeded. 


Diagnosis 

Root  Problems:  How  Joint  Analysis  for  Defense-Planning  Is 
Conceived 

The  roots  of  SSA’s  shortcomings  can  be  understood  by  observing  a 
graphic  (Figure  3.2),  which  simplifies  but  explains  what  SSA  is  sup¬ 
posed  to  do  and  in  fact  did  do  before  dissolution  of  its  campaign¬ 
modeling  capability  (see  Appendix  B).  In  2011,  the  process  began  with 
strategy  and  was  followed  by  OUSDP-led  development  of  defense  plan¬ 
ning  scenarios  expressed  as  rich  essays  depicting  intentions  of  strategy 
and  the  complex  challenges  of  executing  it.  The  essays  included  dilem¬ 
mas  and  uncertainties;  they  painted  a  challenge  space ,  not  a  point.  The 
Joint  Staff  then  developed  concepts  of  operations  to  execute  the  strat¬ 
egy  for  the  broad  scenario  painted  by  the  defense  planning  scenarios.9 
That  was  followed  by  collaborative  studies  of  the  campaign-modeling 
variety  to  establish  enough  context  to  allow  specification  of  the  myriad 
inputs  used  by  campaign  models.  In  this  2011  depiction,  this  effort  was 
led  by  CAPE’s  Simulation  Analysis  Center  (SAC).  An  analytic  baseline 
was  the  full  package  of  scenario,  CONOPS,  context-setting  study,  and 
integrated  joint  data.  The  analytic  baselines  were  intended  to  be  base¬ 
lines  in  the  sense  of  a  common  point  of  departure  for  numerous  and 
substantial  variations  performed  mostly  by  the  services.  The  baselines 
themselves  demonstrated  the  feasibility  of  executing  the  OUSDP- 
provided  strategy,  but  identified  difficulties  in  doing  so  that  were 
itemized  in  massive  documentation  (usually  by  identifying  assump¬ 
tions).  Over  time  (from  year  to  year),  insights  from  one  cycle  informed 
the  next  round,  even  strategy  itself. 

By  2010-2011,  SSA  was  largely  successful  in  carrying  out  this 
process.  I  personally  reviewed  illustrative  SSA  analytic  baselines  and 
was  impressed  by  the  quality  of  the  scenarios,  the  quality  and  docu¬ 
mentation  of  the  CONOPS  (more  properly  referred  to  as  Multi-Service 


Figure  3.2 

Simplified  Depiction  of  the  Support  for  Strategic  Analysis  Process  (Through  2011)  for  Midterm  Planning 


Other  information 

NOTE:  PPBE  =  planning,  programming,  budgeting,  and  executing;  SecDef  =  Secretary  of  Defense;  AoA  =  analysis  of  alternatives. 


RAND  RR1469-3.2 
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Force  Deployments),  and  the  valuable  body  of  knowledge  represented  by 
the  integrated  data.  By  2010-2011,  SSA  was  doing  well  at  what  it  had 
been  tasked  to  do. 

As  of  2011,  CAPE  no  longer  supports  campaign-modeling  and 
no  longer  creates  the  analytic  baselines.  SSA  continues  to  distribute 
older  baselines  and  is  attempting  to  proceed  in  different  ways.  In  par¬ 
ticular,  the  Joint  Staff’s  J-8  has  picked  up  some  of  the  burden  on  an  ad 
hoc  basis,  and  the  services  are  attempting  to  coordinate  more  directly. 
CAPE  and  OUSDP  are  collaborating  to  identify  and  sharpen  key 
issues,  many  of  which  are  mission-level  challenges.  The  situation  is  far 
from  satisfying  to  participants,  but  work  continues  despite  frustrations. 

It  may  seem,  at  first  glance,  that  the  SSA  process  in  Figure  3.2  was 
eminently  sensible,  in  which  case  restoring  all  the  pre-2012  function¬ 
ality  would  seem  to  be  important.  However,  based  on  results  of  this 
study,  I  concluded  otherwise. 

Strategy  is  provided  as  a  given  (the  left  side  of  Figure  3.2).  This 
has  the  effect  of  relegating  analysis  and  support  thereof  to  something 
like  operations  research:  A  problem  is  presented  and  analysts  are  then 
asked  to  show  how  the  problem  can  be  solved  (developing  CONOPS 
and  demonstrating  executability).  Strategy,  however,  does  not  come 
out  of  nowhere  at  the  whim  of  leaders.  Rather,  it  is  a  product  of  initial 
analysis  in  OUSDP.  This  includes  significant  background  research,10  as 
well  as  informal  analysis  that  reconciles  objectives,  feasibility,  and  the 
resources  plausibly  obtainable.  Should  an  SSA-like  activity  not  support 
such  analysis?  And  why  should  such  analysis  not  be  more  explicit?  Why 
should  SSA  only  provide  data  rather  than  do  supporting  analysis? 

Scenarios  are  shown  as  flowing  from  strategy,  but  should  there  be 
explicit  analysis  supporting  the  development  of  scenarios  and  should 
an  SSA-like  activity  be  part  of  it? 

The  step  of  CONOPS  development  requires  the  Joint  Staff  to 
address  a  given  scenario,  develop  an  initial  CONOPS,  and  then  make 
countless  other  assumptions  about  subsequent  operations  and  how 
they  will  turn  out.  Even  if  developers  make  good  choices,  the  result 
is  essentially  the  sketch  of  a  detailed  point  scenario.  Developers  of  the 
CONOPS  are  unable  to  address  all  the  challenges  posed  in  the  scenar¬ 
ios  because  developers  have  to  choose  which  paths  to  take  among  those 
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that  are  possible.  Developers  can  and  do  build  in  some  clever  branch¬ 
ing,  bur  there  are  severe  limitations. 

The  joint  data  and  analytical  baselines  are  even  narrower  in  scope. 
Further,  in  practice,  they  have  been  developed  to  feed  rather-detailed 
campaign  models,  which — as  currently  used — are  not  well  suited  for 
big-picture  thinking  and  broad  uncertainty  analysis.11  Also,  they  are 
not  well  suited  for  addressing  many  of  the  softer  aspects  of  operations, 
including  counterinsurgency,  counterterrorism,  and  stabilization.12 
Thus,  referring  to  these  model  inputs  as  data  fails  to  convey  a  sense  of 
what  may  be  involved. 

The  overall  process  does  not  even  mention  uncertainty  analy¬ 
sis,  although  there  is  some  uncertainty  analysis  used  to  construct  the 
analytical  baselines  within  the  joint  study.  As  noted,  the  assumption 
behind  SSA  was  that  the  services  or  special  joint  studies  would  use  the 
baselines  as  points  of  departure  for  subsequent  broad  exploration,  but, 
in  practice,  such  exploration  was  very  limited — to  the  continued  frus¬ 
tration  of  OUSDP  and  others.  The  baselines  became  end  points.  Also, 
the  baselines  were  sometimes  interpreted  by  the  services  as  specifica¬ 
tions.  This  was  seriously  problematic  because  executability  of  baselines 
may  be  a  necessary  requirement,  but  it  does  not  ensure  the  ability  of  the 
programmed  force  to  deal  with  all  the  many  variants  in  the  original 
(essay-form)  expression  of  scenario.  No  single  point  scenario  can  represent 
the  set  of  necessary  and  sufficient  requirements .13  The  last  point  can  be 
expressed  differently:  The  process  that  has  emphasized  standardization 
of  scenarios,  CONOPS,  and  input  data  for  complex  campaign  models 
has  undercut  the  very  intentions  of  defense  secretaries  seeking  to  build 
capabilities  (forces  and  the  technical  capabilities  they  possess)  that  can 
do  well  for  diverse  situations.  This  was  the  idea  behind  capability-based 
planning,  dating  from  the  mid-1990s  and  reflected  in  the  2001  Qua¬ 
drennial  Defense  Review  (Rumsfeld,  2001).14 

Mission-Level  Versus  Campaign-Level  Analysis 

As  a  related  matter,  CAPE  had  concluded  by  2011  that  many  of  the 
issues  it  sees  as  critical  must  be  studied  on  a  mission-by-mission  level, 
considering  the  many  uncertainties  facing  each  issue.  It  is  the  possible 
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variations  that  matter,  not  the  notional  result  of  any  single  base  case. 
Heavily  standardized  campaign  analysis  had  gotten  in  the  way. 

Those  disagreeing  with  CAPE  argue  that  campaign  analysis  is 
logically  and  practically  necessary:  They  agree  that  having  capabilities 
is  necessary  and  critical,  but  a  key  question  is  whether  the  capabilities 
can  be  put  together  to  accomplish  what  is  called  for  in  strategy.  Can 
the  strategy  be  executed  (a  key  issue  in  force-sizing)?  How  else  can  that 
be  assessed  except  by  campaign  analysis? 

The  synthesis  here  is,  of  course,  that  joint  analysis  needs  to  do 
both.15  Doing  so  requires  recognizing  the  need  for  different  levels  of 
analysis  and  related  levels  of  modeling  (multiresolution  modeling). 
Conducting  a  campaign  is  to  conduct  a  sequence  of  missions  (some 
of  them  at  the  same  time).  Success  of  the  campaign  requires  success  in 
all  of  the  critical  missions.  Campaign  analysis,  however,  need  not  go 
into  the  details  of  how  the  missions  are  accomplished.  Lower-resolution 
campaign  analysis  can  be  (and  was,  in  earlier  decades)  about  ensuring 
the  ability  to  deploy  adequate  forces  to  one  or  multiple  places  in  time 
to  succeed  with  strategy.  What  constitutes  an  adequate  force  may  be 
estimated  from  various  mission-level  analyses  (as  well  as  historical  data 
and  doctrine).  Although  it  is  common  for  DoD  campaign  analyses 
to  have  considerable  detail  (e.g.,  the  war-fighting  detail  that  includes 
attrition,  rates  of  advance,  and  other  dubious  predictions),  that  is  often 
unnecessary  and  even  distracting  for  informing  strategic  decisions. 

Good  People  Can  Partially  Compensate  for  Organizational  Process 
Shortcomings 

Interestingly,  outcomes  of  DoD  processes  (e.g.,  essential  features  of 
QDRs)  are  often  better  than  the  processes  themselves.  This  is  because 
dedicated  people  find  ways  to  do  what  is  most  needed,  despite  short¬ 
comings  of  organization  and  process.  The  product  can  be  good  even 
if  the  process  is  not  always  neat.  Indeed,  currently,  the  actual  process 
is  richer  and  more  subtle  than  graphic  depictions:  Some  of  the  crucial 
thinking  and  analysis  are  accomplished  informally,  whether  in  CAPE, 
OUSDP,  or  the  Joint  Staff.  Nonetheless,  the  remainder  of  this  report 
assumes  that  DoD  would  do  better  if  some  of  that  thinking  and  analy- 
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sis  were  more  explicitly  recognized  and  approached  with  a  reasonable 
degree  of  rigorous  analysis. 

Enduring  Tensions 

Before  turning  to  prescriptions  in  Chapter  Four  it  is  useful  to  summa¬ 
rize  some  enduring  tensions  that  make  defense  analysis,  and  support  of 
joint  analysis  within  defense  analysis,  difficult.  Many  of  these  tensions 
appear  routinely  in  strategic  planning  activities  of  many  organizations. 

Stan  dardiza  tion 

Mature  organizations  like  standardized  processes,  models,  data,  and 
planning  scenarios.  People  typically  want  to  know  what  is  expected  of 
them,  what  they  are  agreeing  to  do,  for  what  they  are  responsible,  and 
for  what  they  will  be  rewarded.  All  this  may,  however,  be  at  odds  with 
what  is  needed  to  deal  effectively  with  change,  uncertainty,  and  dis¬ 
agreement.  The  tension  is  not  unique  to  government  or  DoD,  as  noted 
in  the  academic  literature.16 

There  is  a  clear  need  for  common  data  sets  to  allow  the  compari¬ 
son  of  analyses  using  the  same  assumptions  (a  long-standing  demand  by 
senior  leaders  who  receive  multiple  briefings  that  cannot  be  compared 
directly).  The  tendency,  however,  is  to  never  get  around  to  the  uncer¬ 
tainty  analysis.  The  solution  is  not  to  avoid  baselines  but  to  demand  the 
excursions.  A  stumbling  block  has  been  the  methods  familiar  to  DoD’s 
joint  analysts,  as  discussed  in  Chapter  Four. 

Uncertainty 

Development  of  traditional  defense-analysis  methods  and  models  was 
begun  decades  ago,  when  the  concepts  for  dealing  with  uncertainty 
were  poorly  understood  and  when  it  was  common  to  sweep  them 
under  the  rug.  This  was  an  embarrassing  lapse  noted  even  in  the  1960s 
(Quade  and  Boucher,  1968;  Quade  and  Carter,  1989,  pp.  354-355). 
One  reason  was  an  underlying  tension:  If  one  tried  to  address  uncer¬ 
tainty,  it  was  possible  to  be  overwhelmed  by  complexity  and  to  be  para¬ 
lyzed.  Also,  early  practitioners  sought  scientific  rigor  but  interpreted 
that  to  mean  hard  quantification  and  prediction.  Today,  strong  meth¬ 
ods  exist  for  dealing  effectively  with  uncertainty  without  paralysis. 
Further,  modern  thinking  about  science  and  rigor  requires  dealing  well 
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with  “soft”  uncertainty,  not  wishing  it  away.  To  be  sure,  many  senior 
officials  seek  predictive  analysis  even  when  it  makes  no  sense,  but  such 
officials  as  the  defense  secretaries,  Joint  Staff  chairs,  and  service  chiefs 
are  fully  aware  that  they  are  planning  under  deep  uncertainty.  They 
have  not  been  well  served  by  analysis  that  suppresses  uncertainty. 

Level  of  Resolution  (Level  of  Detail) 

A  tension  always  exists  between  the  desire  to  look  into  detail  and  the 
desire  to  see  the  whole.  Similarly,  there  is  tension  between  seeing  the 
particular  and  seeing  the  realm  of  the  possible.  Organizations  also  rec¬ 
ognize  that  “information  is  power”  and  that  they  may  benefit  by  fram¬ 
ing  issues  in  detail  and  then  controlling  that  detailed  information. 

In  fact,  knowledge  and  analysis  are  needed  at  different  levels  of 
detail.  Senior  leaders  must  focus  at  a  high  level,  but  they  must  rely  on 
those  below  them  to  know  and  deal  with  the  details.  An  SSA-like  activ¬ 
ity  should  provide  such  details  but  also  the  path  to  the  higher-level  rea¬ 
soning.  Policymakers  want  to  understand  the  issues  and  options,  which 
implies  reducing  the  issues  to  manageable  proportions.  Also,  as  stressed 
over  the  years,  the  most-effective  leaders  make  decisions  based  on  gen¬ 
eral  considerations  rather  than  focusing  on  finding  elegant  solutions  to 
very  specific  but  ephemeral  problems  (Drucker,  2006). 

Analysts  dependent  on  computer  models  are  especially  prone  to 
losing  the  forest  for  the  trees.  The  campaign  models  of  DoD  and  analo¬ 
gous  models  used  in  other  domains  involve  tens  of  thousands  of  lines 
of  computer  code,  with  countless  inputs  that  are  a  mix  of  hard  data, 
reasonable  estimates,  and  very  unreliable  assumptions.  Better  exam¬ 
ples  could  hardly  be  found  of  false  precision  than  using  such  complex 
campaign  models  and  imagining  the  results  to  be  predictive,  except 
perhaps  broadly.  All  of  this  implies  a  need  for  multiresolution  research 
and  analysis  and  related  crosscutting  methods,  as  will  be  discussed  in 
Chapter  Four. 

False  Optimization 

Many  analysts  have  been  trained  to  seek  optimal  solutions  for  prob¬ 
lems  posed  by  senior  leaders.  These  analysts  are,  in  a  sense,  solving 
a  math  problem.  Senior  leaders,  however,  recognize  that  this  can  be 
counterproductive  because  of  uncertainties.  Instead,  senior  leaders  seek 
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strategies  that  move  in  the  right  direction,  creating  the  capabilities  for 
flexibility,  adaptiveness,  and  robustness  to  deal  with  realities  as  they 
arise.  To  the  extent  that  the  policymakers  have  utility  functions,  those 
sometimes  emerge  in  the  making  of  decisions  rather  than  being  some¬ 
thing  stable  known  in  advance.  Moreover,  the  utilities  will  change  as 
world  events  occur.  It  follows  that  optimization  can  be  one  of  the  pow¬ 
erful  tools  useful  in  the  depths  of  a  good  analysis  organization  but  not 
as  the  centerpiece  for  resolving  most  senior-leader  issues.17 

Consensus 

Organizations  like  consensus,  as  do  many  senior  officials  and  officers. 
The  search  for  consensus,  however,  can  be  both  paralyzing  and  coun¬ 
terproductive.  This  is  now  a  well-known  problem  in  DoD,  so  it  need 
not  be  elaborated  here.18 


CHAPTER  FOUR 


Prescriptions 


Overview 

Based  on  the  previous  chapter’s  background  of  assessment  and  diagno¬ 
sis,  this  report  recommends  that  DoD  replace  SSA  with  a  new  activ¬ 
ity  rather  than  make  incremental  fixes.  This  is  because  many  of  the 
problems  noted  in  Chapter  Three  are  of  a  fundamental  nature.  In  this 
chapter,  subsequent  sections  deal  with  (1)  reconceiving  the  functions 
of  joint  analysis  and  support  thereof,  (2)  implications  for  organization, 
(3)  implications  for  methods,  and  (4)  implications  for  staffing. 

Some  themes  that  run  through  the  chapter  are  as  follows: 

•  See  strategy  as  an  output  of  analysis  and  deliberation  rather  than 
as  a  stand-alone  input. 

•  Characterize  the  degree  to  which  the  strategy  provides  for  as 
much  flexibility,  adaptiveness  and  robustness  (FARness)  as  possi¬ 
ble  within  budget  constraints.  Also,  characterize  the  implications 
of  smaller  or  larger  budgets  in  such  terms,  thereby  providing  more 
information  about  risk. 

With  these  outputs  in  mind,  it  then  becomes  necessary  to  do  the 
following: 

•  Reconceive  the  questions  asked  of  analysts  and  the  kinds  of  infor¬ 
mation  demanded  from  them. 

•  Adjust  organizations  and  processes  accordingly. 
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•  Broaden  and  change  the  character  of  analytic  methods  and 
research  on  which  to  base  analysis. 

•  Adjust  the  mix  of  staff  accordingly. 


Reconceiving  Functions  and  Attributes  of  Joint  Analysis 

If  the  joint  analytic  activity  is  to  support  strategic  planning,  it  is  appro¬ 
priate  to  consider  what  the  domain  of  such  planning  is.  Figure  4.1  is 
my  version  of  a  diagram  often  shown  by  senior  officials.  When  refer¬ 
ring  to  capability-based  planning  in  the  context  of  strategic  planning, 
another  characterization  sometimes  used  recognizes  several  levels  of 
decision:1 

•  Level  1 — help  senior  leaders  manage  risk  across  the  challenges 
and  determine  the  best  balance  of  investments  across  major  capa¬ 
bility  areas  (e.g.,  QDRs) 


Figure  4.1 

The  Domain  of  Strategic  Planning 
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•  Level  2 — Help  determine  how  best  to  accomplish  missions  or 
joint  concepts  (e.g.,  Joint  Capabilities  Integration  Development 
System  studies) 

•  Level  3 — Help  decide  which  systems  to  buy  or  stop  buying  and 
establish  design  parameters  for  new  systems  (e.g.,  analyses  of 
alternatives). 

In  contrast,  and  as  noted  in  Chapter  Three,  SSA  has  had  only  weak 
contacts  with  technology  and  innovation.  And  SSA  has  not  done  much 
to  support  analysis  of  mission  and  capability-area  issues.  Thus,  SSA  has 
been  supporting  only  some  aspects  of  strategic  planning. 

A  New  Construct  and  Process 

Figure  4.2  shows  a  revised  construct  for  analysis  to  assist  in  mid-  and 
longer-term  planning.1 2  The  lightly  and  more  darkly  shaded  portions 
show  the  new  activity,  which  reflects  the  report’s  suggested  objectives 
for  planning  midterm  capabilities. 

Against  this  background,  this  report  recommends  that  DoD 
create  a  new  activity  to  replace  SSA.  It  could  be  called  analytic  support 
for  strategic  planning  (ASSP). 

1.  ASSP  should  support  initial,  interim  decisions  on  defense  strat¬ 
egy  and  budgets  with  explicit,  understandable,  and  therefore 
relatively  simple  analysis. 

2.  ASSP  should  emphasize  planning  for  flexibility,  adaptiveness, 
and  robustness  (FARness);  ASSP  should  deemphasize  detailed 
analytic  baselines.  As  occurs  now,  it  should  develop  a  list  of  type 
scenarios  (e.g.,  defeat  a  particular  rogue  state  that  is  attacking 
an  ally,  prevail  in  two  simultaneous  wars,  and  cope  adequately 
with  even  more-complex  cases).3  Then,  for  each,  ASSP  should 
develop  spanning  sets  of  variations  to  stress  U.S.  capabilities  in 
all  the  dimensions  needed.  The  spanning-set  scenarios  should 
constitute  necessary  and  sufficient  requirements  for  the  services 
to  meet  in  their  program  development. 

3.  ASSP  should  include  options  that  incorporate  emerging  technol¬ 

ogy  and  innovative  concepts.4  To  assist  in  doing  so,  DoD  should 


Figure  4.2 

Process  for  Analytic  Support  for  Strategic  Planning 
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elevate  the  role  of  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  (USD[AT&L])  and 
have  the  Joint  Staff  be  more  active  in  ensuring  the  surfacing  of 
innovative  service  options  (not  just  programs  of  record  and  tra¬ 
ditional  CONOPS).5 

4.  ASSP  should  help  OSD  focus  on  strategic  considerations6  and 
relatively  low-resolution  analysis  when  establishing  planning 
scenarios  or  studying  mission-level  issues  (also  called  capability 
issues),  with  the  Joint  Staff  having  the  primary  role  for  the  next 
level  of  detail  that  is  especially  important  to  the  services. 

5.  DoD  should  require  the  Joint  Staff  to  be  more  active  in  review¬ 
ing  and  critiquing  service  programs  and  requirement  estimate. 

6.  ASSP  should  change  the  mix  of  analytic  methods,  increasing 
the  emphasis  on  lower-resolution  analysis  with  relatively  simple 
qualitative  models,  quantitative  models,  and  human  wargam¬ 
ing.  These  should  be  sound  abstractions  from  more-detailed 
(higher-resolution)  work.  ASSP  should  apply  this  mixed-method 
approach  to  both  campaign-  and  mission-level  analyses. 

Early  Decisions  with  Spanning  Sets  as  Outputs 

The  first  step  in  Figure  4.2  is  to  inform  initial  decisions  about  strategy 
and  budget.  The  primary  issue  is  what  challenges  the  United  States 
will  take  on  in  developing  its  defense  program.  The  more  challenges 
the  United  States  takes  on,  and  the  higher  the  goals  set  for  each,  the 
higher  the  budget  that  is  necessary.  Achieving  some  goals  is  implau¬ 
sible  even  with  very  high  budgets.  Informing  such  decisions  requires 
analysis  that  is  understandable,  broad  in  scope,  and  helpful  for  making 
strategic  choices  despite  deep  uncertainty.  Such  choices  balance  across 
objectives  and  risks. 

If  the  decisions  determine  requirements  for  the  military  services, 
what  requirements  are  needed,  and  how  should  they  be  expressed?  Using 
concrete  scenarios  is  traditional  for  DoD.  The  concreteness  encourages 
enthusiastic  problem-solving,  in-depth  thinking,  and  communication. 
The  issue  is  deciding  on  which  and  what  kind  of  scenarios.  In  the  mid- 
1990s,  it  was  a  major  and  welcome  change  when  DoD  began  using  a 
number  of  type  scenarios  (e.g.,  defeating  a  regional  rogue,  defeating  a 
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near-peer  regional  power,  or  simultaneous  conflicts).7  DoD  has  since 
added  combination  challenges  (e.g.,  dealing  with  attacks  on  the  home¬ 
land  while  being  engaged  abroad  in  two  theaters).8  Each  such  type 
scenario  could  arise  in  diverse  ways,  demanding  different  U.S.  capabili¬ 
ties.  A  principle  for  evaluating  a  force-planning  strategy  is  that  analysis 
must  cover  the  variations:  Having  the  capability  to  do  well  across  the 
range  of  reasonable  possibilities  matters,  rather  than  being  able  to  do 
very  well  for  a  single  base  case.  The  principle  has  been  expressed  by  sec¬ 
retaries  of  defense  for  many  years.9 

As  demonstrated  in  modern  work,10  coherently  dealing  with  the 
variations  can  be  accomplished  with  a  well-chosen  spanning  set  of  sce¬ 
narios;  capability  to  deal  with  those  scenarios  implies  capability  to  deal 
with  in-between  or  lesser-included  possibilities.  This  is  not  planning  for 
the  worst  case  because  no  such  worst  case  exists.  A  case  that  is  the  worst 
in  some  respects  (e.g.,  short  warning)  will  be  easy  in  others  (e.g.,  the 
adversary  will  also  have  fewer  forces  than  after  a  mobilization).  Conse¬ 
quently,  if  requirements  are  to  be  expressed  as  test-case  scenarios,  many 
such  cases  (a  spanning  set)  are  needed  for  each  type  scenario.  Defining 
these  is  straightforward  for  analysts  but  not  so  for  committees.  Sys¬ 
tematizing  such  work  will  be  a  natural  but  important  extension  of  the 
analysis  for  the  2010  QDR,  which  developed  different  spanning  sets 
to  test  overall  force  structure  by  stressing  the  forces  in  different  ways. 

Figure  4.3  contrasts  old  and  new  approaches.  Under  SSA,  an 
analytic  baseline  was  a  single  detailed  example  (the  ovals)  of  a  type 
scenario  or  challenge.  In  the  new  approach,  each  such  challenge  type 
is  represented  by  a  set  of  low-resolution  cases  (the  circles),  presenting 
different  stresses  for  U.S.  capabilities.11  Because  the  cases  are  low  reso¬ 
lution  in  character,  all  of  them  can  be  examined  analytically  with  suit¬ 
able  models,  including  parametric  variations.  As  illustrated  schemati¬ 
cally  on  the  right  of  Figure  4.3,  results  of  such  exploratory  analysis 
can  be  presented  in  capability  maps  showing  for  which  cases  capabili¬ 
ties  are  sufficient  (the  green  region  means  success  expected),  risky  (the 
yellow  region  means  perhaps  adequate  for  deterrence  at  best),  or  inad¬ 
equate  (red  region  means  corresponding  to  probable  failure).  Good 
analysts  can  develop  these  maps  based  on  broad  considerations  and 
low-resolution  models;  analysts  can  then  identify  test  cases  (indicated 
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Figure  4.3 

Single  Type  Scenarios,  Spanning  Sets,  and  Capability  Maps 
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by  bullets)  that  stress  the  future  force  in  the  different  ways  needed  and 
that  constitute  the  spanning  set.  Those  sets  can  be  studied  in  more 
detail,  improving  and  enriching  the  capability  maps.  These  selective, 
detailed  looks  are  sometimes  crucial. 

Why  is  this  important?  Why  should  DoD  not  just  have  one  illus¬ 
trative  scenario  of  each  class  and  assume  that  the  services  will  build 
capabilities  for  all  the  variants?  Arguably,  during  the  Cold  War,  U.S. 
defense  budgets  were  large,  prices  were  lower,  and  the  services  did 
include  just-in-case  capabilities  and  slack.  That  is  no  longer  the  case 
given  the  budgetary  strains  on  the  U.S.  military  currently  noted  with 
alarm  in  the  2014  QDR  and  its  critique  (Hagel,  2014;  Perry  and  Abi- 
zaid,  2014).  Nonrequired  capabilities  and  slack  are  seen  as  luxuries  as 
the  services  scrimp  in  developing  their  programs  while  also  attempt¬ 
ing  to  protect  such  organizationally  favored  features  as  end  strength, 
number  of  traditional  units,  and  buy  levels  for  programs  of  record. 
Thus,  DoD  needs  to  be  more  explicit  about  the  breadth  of  require¬ 
ments,  or  some  needs  will  not  be  met. 

Although  the  colored  capability  map  in  Figure  4.3  is  simplified 
and  has  only  two  dimensions  of  uncertainty  (preparation  time  and 
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adversary  strength),  the  basis  for  more-complex  uncertainty  analysis 
has  been  laid  over  with  concrete  examples.  The  approach  is  not  hypo¬ 
thetical  and  can  be  comprehensibly  extended  to  consider  many  more 
low-resolution  dimensions  of  uncertainty  (e.g.,  aggregate-level  charac¬ 
terizations  of  the  behavior  of  U.S.  allies  and  adversaries,  the  operational 
strategies  adopted,  the  real-world  effectiveness  of  new  weapon  systems, 
and  the  effect  of  such  shocks  as  initial  cyber  attacks  on  command  and 
control).  Although  now  well  demonstrated,  such  analysis  requires  dif¬ 
ferent  methods  and  models  than  those  used  in  the  SSA  activity,  as  well 
as  people  skilled  in  using  the  new  methods  and  models. 

Balancing  Mission-  and  Campaign-Level  Analyses 

Another  objective  for  ASSP  is  elevating  the  relative  emphasis  on  mis¬ 
sion-level  work  (also  called  capability-area  work).  Many  crucial  defense¬ 
planning  issues  are  about  how  to  achieve  capabilities  for  accomplishing 
such  increasingly  difficult  missions  as  defeating  integrated  air  defenses. 
This  is  evident  from  even  a  cursory  skim  of  recent  QDRs.  The  ASSP 
activity  should  achieve  a  balance  that  gives  at  least  as  much,  if  not  more, 
attention  to  mission-level  issues.  OUSDP  and  CAPE  should  primarily 
be  concerned  with  higher-level  (lower-resolution)  aspects  of  both,  while 
the  services  must  address  such  matters  in  more  depth.  Precisely  the 
same  methods  as  described  above  apply  well  to  mission-level  analysis. 

Despite  the  challenges  in  doing  so,  ASSP  should  also  achieve  a 
balance  with  respect  to  types  of  conflict.  In  particular,  ASSP  must  give 
appropriate  weight  to  analysis  relating  to  counterterrorism,  stabiliza¬ 
tion,  and  irregular  warfare.  The  methods  for  studying  these  matters 
should  draw  on  social  science  (and,  in  some  cases,  be  the  methods  of 
social  science). 


Implications  for  Organization 

Participation 

ASSP  should  have  the  same  membership  as  the  previous  analytic  agenda 
and  SSA  activities  (see  Appendix  B):  OUSDP,  CAPE,  USD(AT&L), 
the  Joint  Staff,  the  services,  and  others  as  needed.  However,  some  activ¬ 
ities,  particularly  those  informing  interim  decisions  regarding  strat- 
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egy  and  defense  scenarios  while  accounting  for  resource  constraints, 
will  continue  to  be  more  tightly  limited  to  OSD  and  the  Joint  Staff. 
Whether  those  activities  should  be  regarded  as  part  of  ASSP  is  a  sepa¬ 
rate  decision. 


Responsibilities 
Functional  Issues 

As  summarized  in  Table  4.1,  the  new  construct  indicated  in  Figure  4.2 
changes  the  responsibilities  of  DoD  components.  The  first  step  of 
developing  strategy  and  defense-planning  scenarios  is  replaced  by  the 
larger  step  of  first  informing  interim  decisions  on  strategy  and  budget. 
This  should  be  accomplished  by  an  elite  team  or  task  force  (not  a  com¬ 
mittee)  led  by  CAPE  and  OUSDP — CAPE  because  the  effort  requires 
rigorous,  low-resolution  analysis  involving  capabilities,  costs,  and 
trade-offs  and  OUSDP  because  the  effort  is  about  establishing  strategy. 


Table  4.1 

Responsibilities  in  the  Analysis  for  Support  of  Strategic  Planning  Activity 


Inform  Initial 
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NOTES:  The  task  force  activity  (left  side)  might  or  might  not  be  considered  part  of 
ASSP  per  se.  Number  of  bullets  indicates  relative  responsibility.  SecDef  =  Secretary 
of  Defense;  VCJS  =  Vice  Chairman,  Joint  Staff;  J5  =  Director  for  Strategic  Plans  and 
Policy,  Joint  Staff;  J8  =  Director,  Force  Structure,  Resources,  and  Assessment,  Joint 
Staff. 
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DoD  should  consider  having  some  top-quality  analysts  functionally 
dual-hatted  for  CAPE  and  OUSDP  (see  below).  Representatives  from 
USD(AT&L)  and  the  Joint  Staff  should  ensure  attention  to  innovation 
and  military  feasibility.  Results  of  this  crucial  early  effort  should  be 
approved  by  the  Secretary  of  Defense  and  (ultimately)  the  President. 
To  be  sure,  such  an  organizational  change  might  be  difficult,  but  the 
proposal  has  precedent. 

The  next  step  shown  in  the  table  is  that  OUSDP  and  the  Joint 
Staff  articulate  strategy  and  develop  CONOPS.  The  last  two  columns 
are  new,  corresponding  to  the  lightly  and  more  darkly  shaded  regions 
of  Figure  4.1.  The  more-strategic,  low-resolution  activities  are  primarily 
the  responsibility  of  CAPE,  with  the  Secretary  of  Defense  approving 
the  final  expression  of  requirements;  those  of  a  more  detailed  nature  are 
primarily  the  responsibility  of  the  Joint  Staff.  Although  all  ASSP  con¬ 
tributors  would  review  and  comment  on  everything,  as  occurs  within 
SSA,  Table  4.1  indicates  with  two  bullets  that  it  is  especially  important 
for  the  Joint  Staff  (both  J-5  and  J-8)  and  USD(AT&L)  to  be  involved 
in  the  low-resolution  refining  and  that  CAPE  and  USD(AT&L)  should 
be  involved  in  the  more  detailed  work.  The  goal  should  not  be  consen¬ 
sus  but  ensuring  quality  and  innovation. 

One  aspect  of  the  new  construct  worth  noting  is  that  the  later  por¬ 
tions  of  the  ASSP  process  (the  part  most  akin  to  the  current  SSA)  would 
have  new  responsibilities  for  refining  capability  maps  and  the  related 
matter  of  risk  assessment.  In  a  sense,  SSA’s  detailed  analytic  baselines 
for  point  cases  would  be  replaced  by  the  ASSP’s  lower-resolution  capa¬ 
bility  maps. 

Dealing  with  Organizational  Tensions 

How  might  the  CAPE-OUSDP  task  force  shown  in  Figure  4.2  be 
achieved?  Possibilities  include  having  a  group  within  OUSDP  effec¬ 
tively  dual-hatted,  responsive  to  and  working  closely  with  both  the 
Under  Secretary  of  Defense  for  Policy  and  the  director  of  CAPE; 
having  a  group  within  CAPE  effectively  dual-hatted  in  the  same  way; 
or  having  the  task  force  report  directly  to  the  deputy  secretary. 
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Locus  in  OUSDP 

It  might  be  argued  that  the  basis  for  the  first  organization  already 
exists  in  OUSDP  as  the  Assistant  Secretary  for  Strategy,  Plans  and 
Capabilities  (particularly  the  Strategy  office  beneath  it).  That  office  (see 
Figure  4.4),  however,  does  not  currently  see  itself  as  dual-hatted  or  as 
serving  functions  in  both  the  OUSDP  and  CAPE  lines. 

Locus  in  CAPE 

In  earlier  decades,  the  predecessor  of  CAPE,  Program  Analysis  and 
Evaluation  (PA&E),  had  a  deputate  called  Regional  Programs,  which 
routinely  contributed  to  the  functions  of  OUSDP  and  PA&E.  Regional 
Programs  generated  the  defense-planning  scenarios  and  influenced 
programs.  The  deputate ’s  analysts  were  seen  as  both  program  analysts 
and  strategic  analysts.  In  the  1990s,  DoD  reorganized  and  created  the 
ASD  for  Strategy  and  Resources,  the  predecessor  of  today’s  Assistant 
Secretary  of  Defense  (ASD)  for  Strategy,  Plans  and  Capabilities.  The 
office  took  on  responsibilities  for  building  strategy  and  related  scenar¬ 
ios.  In  my  observation,  the  organizational  transition,  which  seemed 
logical  at  the  time,  was  not  fully  successful  because  the  office  had  never 
been  very  influential  in  program  development.12  Nor  did  it  have  a  suf¬ 
ficient  cadre  of  analysts  respected  across  DoD  for  program  analysis  to 
complement  its  expertise  in  strategy  analysis. 

Figure  4.4 
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Locus  in  OSD 

It  might  be  that  the  dual-hatting  would  not  work  or  that  CAPE  would 
be  unwilling  to  reassume  the  responsibility  for  leading  DoD-wide 
analysis.  CAPE  perhaps  has  this  responsibility  under  its  charter,  but 
that  responsibility  is  the  third  priority,  well  behind  program  evalua¬ 
tion  and  cost  assessment.  During  my  research  for  this  study,  observers 
I  spoke  with  noted  that  CAPE  no  longer  supports  or  participates  in  the 
professional  society  promoting  military  operations  research  and  seems 
to  have  little  interest  in  leading  DoD-wide  analysis.  That  could  change, 
especially  if  necessary  billets  were  added,  but  the  last  option  would  be 
to  have  the  functionally  dual-hatted  organization  report  to  the  deputy 
secretary.  That  would  be  natural  in  some  respects  and  might  work 
well  with  some  combinations  of  personalities,  but  it  would  expand  the 
office’s  already  large  span  of  control  and  probably  cause  the  organiza¬ 
tion  to  have  a  tense  relationship  with  both  OUSDP  and  CAPE.13 

A  Portfolio  of  Supporting  Activities 

The  previous  section  discussed  organizational  structure  at  an  aggregate 
level:  what  offices  must  be  involved,  who  should  have  what  responsi¬ 
bilities,  and  how  information-sharing  should  be  achieved.  A  separate 
issue  is  the  portfolio  of  activities.  Historically,  the  analytic  agenda  and 
SSA  were  dominated  by  campaign-modeling,  with  some  important 
exceptions. 

In  light  of  the  discussion  in  Chapter  Three  on  methods  and  long¬ 
standing  shortcomings,  the  recommendation  here  is  that  DoD  should 
conceive  of  its  ASSP  analytic  activities  in  portfolio  terms,  as  suggested 
in  Figure  4.5. 

This  approach  would  push  back  against  the  common  tendency 
of  analytical  organizations  to  focus  on  increasingly  detailed  computer 
models  over  time  with  a  continuing  and  expensive  search  for  agreed- 
on  data  and  refinements,  while  putting  little  investment  into  research, 
multiresolution  modeling,  or  integration  of  knowledge  across  sources 
and  levels. 

On  the  one  hand,  an  organized  approach  such  as  that  suggested 
for  ASSP  would  fill  in  gaps  and  provide  integrated  analytic  capabili¬ 
ties.  It  is  also  possible,  however,  that  it  would  crowd  out  ad  hoc  proj- 
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Figure  4.5 

An  Activity  Domain  for  Supporting  Activities 
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ects  by  various  DoD  components  that  might  be  of  higher  quality  and 
more  directly  relevant  than  those  produced  by  a  centralized  activity. 
From  time  to  time,  DoD  officials  have  attempted  to  centralize  and 
systematize  modeling;  the  results  have  typically  been  bad.  Thus,  any 
structure  such  as  that  displayed  in  Figure  4.5  should  insist  on  diversity 
and  competition,  rather  than  pursuit  of  some  allegedly  cost-effective 
standard.  Research,  modeling,  and  analysis  are  not  like  commodities 
except  perhaps  when  the  phenomena  they  deal  with  are  well  under¬ 
stood  and  stable. 


Implications  for  Methods  and  Tools 

Legacy  Methods  and  the  Broader  Landscape 

Campaign-modeling  dominated  SSA  until  2011.  Some  human 
wargaming  has  been  added  in  recent  years.  The  spectrum  of  relevant 
methods  is  actually  much  broader.  Table  4.2  is  one  version  of  a  table 
that  appeared  in  reports  and  briefings  over  time.14  The  instruments 
range  from  simple  back-of-the-envelope  models  to  field  experiments. 
Although  incomplete,  the  list  broadens  the  scope  of  thinking  about 
what  tools  the  analyst  can  use.  The  columns  describe  various  attri- 


Table  4.2 

Illustrating  the  Broad  Range  of  Analytic  Instruments 


Relative  Strength 


Strategic-Level  Functionality 
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Strategic  Physical 
Integration  Phenomena 


Human 

Phenomena 


Simple  analytical8  Low 

Seminar-level  human  wargaming  Low 

Red-teaming  on  capabilities  and  operations  Varied 

Qualitative  factor  trees  Low 

Human  wargaming  Medium 

Campaign  simulation  (usual)  Medium 

+  agents,  political-economic  factors.  Medium 

and  exploratory  analysis^ 

Mission-level  adaptive  models,  exploratory  Medium 

analysis 

High-fidelity  simulation0  High 

Historical  case  studies  Varied 

Historical  data  analysis  Low 

Field  experiments  and  war  data  Varied 


Empirical 

Cautions 


SOURCE:  Davis,  2014,  Table  3.1. 

NOTES:  Ratings  are  1  (very  poor)  to  5  (very  good),  with  red,  orange,  yellow,  light  green,  and  green  corresponding  to  1,  2,  3,  4,  and  5, 
respectively.  Scores  depend  on  assumptions. 

8  Examples  include  closed-form  models  and  spreadsheet-level  computer  models. 

k  Exploratory  analysis  examines  the  effect  of  simultaneous  variations  of  all  important  assumptions,  not  mere  sensitivity  analysis  on  the  margin. 
c  In  some  instances,  high-fidelity  simulation  can  be  a  primary  and  reliable  source  of  what  can  be  considered  to  be  empirical  information.  It  is 
simply  not  feasible  to  obtain  the  equivalent  information  with  physical  testing. 
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butes  of  the  methods.  Although  the  scoring  is  subjective  and  depen¬ 
dent  on  detailed  assumptions  (see  Davis,  2014),  the  primary  point  is 
that  some  methods  tend  to  be  associated  with  low  resolution,  agility, 
creativity,  and  transparency  (top  left),  while  others  are  associated  with 
reality  (bottom  right).  No  single  method  is  good  across  the  board,  and, 
to  make  things  worse,  typical  campaign  simulations  have  been  much 
weaker  in  important  attributes  than  they  need  to  be. 

DoD  should  arguably  be  using  all  of  these  methods  and  sources 
of  information,  as  well  as  some  not  on  the  list,  as  contributors  to  ASSP. 
We  ignore  history  at  our  peril,  as  was  demonstrated  in  2003,  when 
U.S.  forces  employed  in  Iraq  proved  inadequate  for  the  insurgency  that 
arose.  History-based  analysis  had  pointed  out  the  potential  need  for 
much  larger  forces,  but  this  idea  had  been  disparaged  or  sloughed  off. 
As  a  second  example,  the  asymmetric  tactics  being  used  by  the  Islamic 
State  of  Iraq  and  the  Levant  (ISIL)  and  other  adversaries  today  have 
come  as  no  surprise  to  history-minded  analysts  but  were  not  antici¬ 
pated  in  campaign  studies  of  the  traditional  variety.  A  contrasting 
example:  In  the  1990s,  I  remember  well  campaign  analyses  that  used 
history-based  parameter  values  to  determine  the  effectiveness  of  air 
forces  grossly  underestimated  the  effectiveness  of  air  forces  in  the  age 
of  precision  fires,  whereas  other  simple  models  coupled  with  test  data 
proved  prescient.15 

At  the  higher  levels  of  political-military  analysis,  seminar 
wargames  and  strategic  red-teaming  have  a  long  history  of  anticipating 
developments  (and,  to  be  sure,  sometimes  those  developments  did  not 
materialize)  (Defense  Science  Board,  2003b).  This  is  among  the  many 
reasons  for  the  resurgence  of  interest  in  wargaming. 

An  entirely  different  class  of  analysis  has  arisen  in  response  to  the 
September  11,  2001,  attacks  by  al-Qaeda  and  the  subsequent  global 
war  on  terrorism.  Some  of  the  tools  have  been  variants  of  campaign 
models  that  include  certain  political-military  aspects  related  to,  for 
example,  insurgency  and  its  roots  in  societal  discontent  (Body  and  Mar- 
ston,  2011).  Others  have  been  mainstream  social  science  tools,  such  as 
methods  for  eliciting  expert  opinion,  quantitative  correlational  work, 
forecasting,  and  case  studies  (see,  e.g.,  Wong,  2014).  Still  others  have 
contributed  to  an  evolving  approach  to  causal  social  science  modeling 
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about,  for  example,  the  factors  contributing  to  terrorism  and  public 
support  thereof.  That  work  was  sponsored  by  the  DoD  analytic  com¬ 
munity  (many  of  the  same  people  associated  with  SSA)  and  drew  heav¬ 
ily  on  the  academic  social  science  literature  and  major  contributors  to 
that  literature  to  integrate  many  of  the  seemingly  inconsistent  threads 
of  research  on  counterterrorism.16  In  more-recent  times,  this  work  has 
been  extended  to  generate  prototypes  of  computational  models  that  are 
based  on  qualitative  social  science  research  but  designed  to  highlight 
uncertainties  and  facilitate  analysis  in  inherently  “soft”  circumstances.17 

Uncertainty  Analysis 

A  theme  throughout  this  report  is  DoD’s  need  to  do  far  better  in 
addressing  uncertainty,  and  to  do  so  routinely  in  analysis  rather  than 
by  exception.  A  great  deal  of  modern  research  describes  methods  for 
doing  so,  with  numerous  published  examples  in  a  variety  of  social- 
policy  and  national-security  domains.  Such  uncertainty  analysis 
was  not  feasible  decades  ago  when  the  legacy  analytic  methods  were 
developed,  but  much  progress  has  occurred.  A  few  key  points  are  the 
following: 

•  The  methods  apply  not  just  to  “normal”  uncertainty  (as  when  rep¬ 
resenting  empirical  data  on  reliability  statistics)  but  to  deep  uncer¬ 
tainty,  which  has  been  referred  to  over  the  years  as,  for  exam¬ 
ple,  real  uncertainty,  scenario  uncertainty,  and  possible  wild  cards. 
More  technically,  deep  uncertainty  has  been  defined  as  the  condi¬ 
tion  in  which  “the  parties  to  a  decision  do  not  know  or  agree  on 
the  system  models  relating  actions  to  consequences  or  the  prior 
probability  distributions  for  the  key  input  parameter  to  those 
models.”18  That  situation  is  the  norm  in  strategic  planning. 

•  Addressing  deep  uncertainty  with  modern  methods  can  aid  in 
practical  decisionmaking,  rather  than  lead  to  the  paralysis  by 
analysis  deplored  by  decisionmakers.19  Such  methods  relate  well  to 
DoD’s  needs  for  capabilities  analysis  and  strategic  portfolio  analy¬ 
sis.  They  work  best,  however,  when  used  with  relatively  simple 
models  or  multiresolution  families  of  models  (Davis,  Gompert, 
et  ah,  2008;  Davis,  2014). 
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Human  Wargaming 

In  May  2015,  Deputy  Secretary  of  Defense  Robert  Work  announced  a 
new  initiative  to  use  human  wargaming  as  a  major  tool  in  contemplat¬ 
ing  needs  for  the  years  ahead,20  distinguishing  among  roughly  three 
time  horizons:  the  first  Future  Years  Defense  Program  (FYDP),  the 
second  and  third  FYDP,  and  the  fourth  FYDP  and  beyond.  One  part  of 
this  was  to  “reinvigorate  Support  for  Strategic  Analysis  (SSA)”  with  the 
intention  that  OUSDP  will  conduct  wargames  to  inform  SSA  scenario 
development,  that  the  Joint  Staff  will  conduct  wargames  to  inform 
CONOPS  and  force  development,  and  that  CAPE  will  “manage  the 
development  of  force  capability  and  capability  excursions  in  SSA  sce¬ 
narios  to  allow  a  broader  exploration  of  risks  and  solutions.”21 

Since  the  announcement,  a  great  deal  of  activity  ensued  as  indi¬ 
viduals  and  organizations  reacquainted  themselves  with  wargaming 
and  contemplated  how  they  could  contribute.  Further,  DoD  compo¬ 
nents  conducted  a  large  number  of  wargames.  Authors  have  written 
thoughtful  papers  based  on  experience,22  as  well  as  earlier  reviews  of 
wargaming.23  The  results  of  one  recent  game  illustrate  timeliness  and 
flexibility  (Shlapak  and  Johnson,  2016).  This  game  addressed  the  chal¬ 
lenges  faced  by  NATO  in  deterring  or  defending  its  Baltic-state  mem¬ 
bers  from  possible  aggression  by  Russia.  It  was  an  interesting  example 
of  how  low-resolution  work  accomplished  in  a  relatively  short  time  can 
rather  convincingly  concentrate  minds,  rule  out  some  options,  and  sug¬ 
gest  a  path  ahead  that  is  not  sensitive  to  the  kinds  of  details  that  must 
be  dealt  with  in  computer  models.  Such  games  often  set  the  stage  for 
subsequent  work,  as  illustrated  by  several  decades  of  experience  with 
RAND’s  Day  After  wargames.24 

An  enduring  challenge  is  how  to  use  human  gaming  more  rigor¬ 
ously,  enough  so  that  gaming  can  be  seen  as  part  of  the  analytic  process. 
Views  differ,  but  Figure  4.6  indicates  schematically  the  approach  high¬ 
lighted  in  National  Research  Council,  2014. 25  This  approach  builds 
cumulative  knowledge  into  models,  which  can  then  be  used  for  trans¬ 
parent,  reproducible,  rigorous  analysis.  Gaming  is  a  key  mechanism 
for  building  the  knowledge:  identifying  factors  that  human  players 
immediately  recognize  but  that  initial  modeling  has  omitted,  gaining 
a  better  sense  of  how  humans  might  act,  and  so  on.  This  is  a  model- 
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Figure  4.6 

Integrating  Human  Wargaming  and  Modeling 
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test-model  approach.26  The  model  informs  game  design;  the  game 
results  inform  model  improvement.  The  example  assumes  an  applica¬ 
tion  studying  issues  of  deterrence  and  escalation,  such  as  in  a  possible 
conflict  with  North  Korea.  The  model  referred  to  is  a  simple  cognitive 
model  of  North  Korea,  one  that  attempts  to  represent  the  factors  that 
North  Korean  leaders  would  consider  when  contemplating  escalation 
or  deescalation  (see  National  Research  Council,  2014). 

A  very  different  approach  regards  gaming  as  involving  low- 
resolution  models  adequate  for  certain  important  purposes,  espe¬ 
cially  when  results  are  overdetermined.  The  recent  gaming  on  deter¬ 
ring  Russian  aggression  in  the  Baltics  is  an  example  (Shlapak  and 
Johnson,  2016). 

Another  unresolved  problem  is  what  should  constitute  validation 
for  human  gaming.  A  great  deal  of  effort  has  gone  into  studying  and 
writing  about  the  validation  of  models,  but  much  less  has  been  done 
on  this  for  human  gaming.  The  primary  need  is  to  have  first-class  par¬ 
ticipants  and  enough  funding  to  do  things  properly.  Preparing  a  serious 
game  can  and  should  take  significant  time,  whereas  merely  throwing 
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together  a  quick  game  is  deceptively  simple.  This  is  something  with 
which  several  military  institutions  are  well  aware  from  decades  of  expe¬ 
rience  (e.g.,  the  U.S.  Naval  War  College).  The  Joint  Staff  has  also  con¬ 
ducted  numerous  high-level  wargames  for  many  years,  many  of  them 
seminar-style.  In  recent  years,  FFRDCs  have  also  conducted  wargames 
for  several  of  the  DoD  components  on  a  variety  of  current  issues. 

Overall,  the  rediscovery  of  human  wargaming  is  very  much  to  be 
applauded.  How  well  it  fits  into  DoD’s  joint  analysis  activities  is  yet  to 
be  determined.  DoD  is  proposing  significant  efforts  of  this  kind  in  its 
budgeting,  in  part  to  revitalize  the  SSA  activity. 

New  Campaign  and  Mission  Models 

Given  controversy  about  campaign  models,  it  might  seem  odd  to  sug¬ 
gest  new  developments,  but  campaign  analysis  is  crucial  for  integration 
and  for  certain  activities,  such  as  testing  the  executability  and  appro¬ 
priateness  of  a  tentative  strategy  and  associated  programs,  or  identify¬ 
ing  such  problems  as  the  double-counting  of  units  and  subtle  inter¬ 
dependencies.  These  activities  are  useful  in  fore  e-sizing  analysis.  The 
campaign  analyses  conducted  within  DoD  over  the  past  two  to  three 
decades  have  sometimes  been  associated  with  complexity,  opaqueness, 
inflexibility,  and  failure  to  address  uncertainty,  but  none  of  these  short¬ 
comings  is  inherent.  Mission  models  can  be  even  more  detailed  and 
(except  to  those  who  use  them)  opaque.  They  also,  however,  can  be  rel¬ 
atively  simple,  parametric,  and  suitable  for  higher-level  analysis.  They 
are  useful  in  illuminating  what  some  refer  to  as  capability  issues. 

Since  relatively  detailed  campaign  and  mission  models  already 
exist,  a  priority  should  be  for  DoD  to  develop  simpler  models  and 
connections  between  those  and  the  more-detailed  models.  Table  4.3 
sketches  alternative  ways  to  develop  these  simpler  models.  All  have 
precedents  (Davis,  2014,  Chapter  Four).  The  scoring  of  the  options 
depends  on  a  number  of  questionable  assumptions,  so  the  reader  should 
take  away  only  the  fact  that  options  exist.  An  example  of  an  empirical 
model  would  be  CAPE’s  statistical  models  of  system  cost  or  potential 
cost  growth.  Qualitative  models  may  use  graphics  (e.g.,  factor  trees) 
or  logic  tables  showing  the  factors  that  drive  results.  Simple  models, 
started  from  scratch,  are  often  spreadsheet-level  in  complexity.  CAPE 
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Table  4.3 

Alternative  Paths  to  Simple  Models 
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NOTE:  Number  of  bullets  denotes,  roughly,  degree  to  which  option  achieves 
attribute  of  column. 


a  Regressions  may  be  simple  but  not  substantively  revealing. 
b  Policymakers  often  accept  clear  analysis  by  trusted  agents  while  organizations 
relying  on  big  models  resist. 

c  Development  risk  is  highly  variable  and  depends  on  quality  of  team,  clarity  of 
purpose,  absence  of  consensus  culture,  and  so  on. 

d  Highly  dependent  on  the  quality  of  design  and,  when  relating  to  existing  models, 
the  nature  of  the  model. 


is  sponsoring  a  recent  study  that  the  author  regards  as  building  a  low- 
resolution  “campaign  model”  designed  explicitly  for  far-ranging  uncer¬ 
tainty  analysis  at  the  strategic  level.27  Simplifying  parts  of  an  existing 
model  might  mean,  for  example,  turning  a  module  of  a  large  model 
such  as  STORM  into  a  stand-alone  model  and  eliminating  some  of  its 
detail.  A  motivated  metamodel  is  a  statistical  model  fitted  to  a  form 
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suggested  by  an  understanding  of  the  phenomenon  being  viewed, 
rather  than  based  on  linear  regression  of  input  variables.  Building  new 
multi  resolution  models  or  families  is  straightforward  for  relatively 
simple  problems  (e.g.,  modeling  close-air  support’s  effectiveness  as  a 
function  of  a  half-dozen  variables)  but  more  complex  for  something 
such  as  campaign  models.  Such  models  allow  the  user  to  decide  on  the 
level  of  detail  to  be  entered  as  inputs  or  the  level  of  detail  to  reason  at. 

Deciding  which  combination  of  these  approaches  should  be  taken 
goes  far  beyond  the  scope  of  this  report  and  depends  on  developer 
teams,  the  detailed  characteristics  of  the  current  models,  and  DoD’s 
ability  to  manage  work.  Past  experience  on  the  last  item  is  not  encour¬ 
aging.  I  note,  as  corroborated  in  my  interviewing  for  this  report,  that 
the  last  major  DoD  development,  the  Joint  Warfare  System  (JWARS), 
later  called  the  Joint  Analysis  System  (JAS),  is  largely  considered  to 
have  been  a  fiasco.  The  system  is  estimated  to  have  involved  500  person- 
years  of  effort  over  ten  years  and  was  almost  never  used  for  its  intended 
DoD  purposes  (see  Allen  et  ah,  2007,  p.  9).  The  JWARS  development 
had  numerous  problems  from  the  outset.  In  many  respects,  the  system 
was  designed  by  a  committee,  with  the  requirements  stemming  from 
lengthy  wish  lists  from  all  stakeholders.  A  second  problem  was  that 
JWARS  was  postulated  to  serve  many  functions  with  very  different 
needs.  Yet  a  third  problem  was  that,  although  aspects  of  the  software 
development  were  modern,  the  result  lacked  adequate  modularity  and 
multi  resolution  features. 

Other  model  developments,  such  as  those  for  IDAGAM, 
TACWAR,  THUNDER,  JICM,  and  STORM  (the  models  are  known 
by  their  acronyms),  were  developed  by  single  organizations.  The  level  of 
effort  for  the  first  four  was  at  least  20  person-years  but  not  500  (Allen 
et  ah,  2007). 

Campaign  analysis  for  particular  studies  has  often  been  accom¬ 
plished  with  smaller,  simpler,  and  highly  parametric  models  focused 
on  the  particular  issues  of  interest  (e.g.,  strategic  mobility,  attacking 
any  invading  mechanized  armored  forces  with  precision  weapons, 
ballistic-missile  defense,  the  air  campaign  against  a  peer  competitor’s 
fixed  targets).  Such  analysis  is  sometimes  described  as  mission-level 
analysis,  but  it  is  actually  module-focused  campaign  analysis,  still  at 
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relatively  low  resolution  and  highly  parametric.  To  give  one  simple 
example,  for  long-range  precision  fires  to  halt  an  invading  mecha¬ 
nized  force  moving  along  major  roads,  results  can  be  well  understood 
with  simple  models  involving,  for  example,  sorties  per  day,  kills  per 
sortie,  movement  speed  of  the  army,  the  army’s  break  point  (the  level 
of  attrition  beyond  which  movement  can  be  assumed  to  stop),  and  a 
few  other  parameters  (Ochmanek  et  ah,  1998;  Davis,  McEver,  and 
Wilson,  2002). 

It  is  of  interest  that  a  community  review  in  2007  (Davis  and  Hen- 
ninger,  2007)  recommended  a  new  approach  to  theater  modeling  that 
would  emphasize  modularity,  competition  among  module  developers, 
uncertainty  analysis,  and  so  on. 

Analytic  Tools  for  Counterterrorism,  Stabilization,  and  Irregular 
Warfare 

Among  the  more-challenging  issues  for  DoD  joint  analysis  are  those 
posed  by  counterterrorism  and  irregular  warfare.  It  is  unclear  to  what 
extent  such  issues  are  amenable  to  modeling,  but  some  advances  have 
been  made  in  qualitative  modeling  (Davis  and  Cragin,  2009)  and 
even  in  extending  social  science-based  modeling  to  a  special  kind  of 
computational  modeling  that  emphasizes  uncertainty  and  frames  for 
thinking  rather  than  prediction  (Davis  and  O’Mahony,  2013).  Human 
gaming  has  proven  useful. 


Implications  for  Staffing 

It  is  unlikely  that  DoD  will  want  to  do  a  whole-scale  review  of  its  staff¬ 
ing  for  joint  analysis.  Many  excellent  practitioners  are  in  place.  How¬ 
ever,  changes  are  possible  and  desirable.  DoD  should  plan  simultane¬ 
ously  for  a  mix  of  in-house  and  FFRDC  capabilities  (with  additional 
but  crucial  capabilities  in  laboratories  and  industry).  Appendix  D  dis¬ 
cusses  briefly  some  options  for  DoD  to  consider  on  the  specific  matter 
of  where  to  develop,  maintain,  and  do  joint  campaign  and  mission- 
level  modeling. 
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Misconceptions 
Who  Are  Analysts? 

A  parochial  interpretation  of  the  term  analyst  is  that  analysts  do  what¬ 
ever  it  is  that  CAPE  and  corresponding  component  offices  do,  which 
is  sometimes  referred  to  as  number-crunching,  running  big  models, 
and  budgeting.  In  fact,  analysts  are  distributed  throughout  DoD  and 
related  contract  organizations.  Analysts  perform  many  very  different 
kinds  of  functions.  For  example,  USD(AT&L)  and  OSD  for  Personnel 
and  Readiness  need  a  great  deal  of  analysis,  as  does  OUSDP,  although 
its  analysis  is  less  quantitative  in  nature.  Within  the  Joint  Staff,  analysis 
occurs  in  many  of  the  offices. 

Even  with  CAPE,  most  analysts  are  not  the  analysts  associated 
with  the  SSA  activity.  Instead,  they  work  in  particular  divisions  deal¬ 
ing  with,  for  example,  ground,  air,  or  land  forces;  strategic  forces;  or 
cyber  war — usually  at  the  level  of  specific  programs.  Somewhat  the 
same  is  true,  but  with  important  differences,  for  analysts  in  Air  Force 
Studies  and  Analysis,  the  Navy’s  N-81,  the  Marine  Corps  Combat 
Development  Command,  and  the  Army’s  G-8  and  Center  for  Army 
Analysis  offices.28  Many  of  the  analysts  outside  the  SSA  activity  at  least 
think  and  work  in  the  joint  context,  if  not  always  with  an  appreciation 
of  subtleties. 

What  Are  the  Backgrounds  of  Good  Analysts? 

Even  when  referring  to  the  DoD  joint-analysis  domain,  the  term  analyst 
should  not  be  interpreted  as  synonymous  with  someone  with  an  opera¬ 
tions  research  degree.  It  is  merely  an  artifact  of  bureaucratic  processes 
over  time  that  this  misconception  has  arisen  (services  assign  specialty 
codes  to  their  personnel,  based  in  part  on  degrees  achieved,  and  opera¬ 
tions  research  is  one  of  them).  Taking  a  broader  view,  top-notch  ana¬ 
lysts — even  those  capable  of  quantitative  or  otherwise  rigorous  analy¬ 
sis — have  been  educated  in  a  broad  range  of  disciplines,  including  the 
physical  sciences,  economics,  engineering,  mathematics,  applied  math¬ 
ematics  (e.g.,  operations  research),  computer  science,  law,  and  politi¬ 
cal  science.  Indeed,  those  often  identified  as  giants  in  the  pantheon 
of  past  DoD  analysts  have  only  sometimes  been  trained  in  operations 
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research.  Appendix  C  discusses  backgrounds  of  notable  analysts  and  of 
those  hired  in  some  analysis  institutions.  The  primary  conclusion  is  one 
of  diversity  in  disciplinary  background. 


CHAPTER  FIVE 


Recommendations 


Much  of  Chapter  Four  amounts  to  recommendations,  but  it  is  well 
to  synopsize  the  recommendations  here.  This  report  recommends  that 
DoD  do  the  following: 

•  Adopt  a  new  planning  construct  (see  Figure  4.2)  with  new  orga¬ 
nizational  responsibilities  (see  Table  4.1). 

•  Revamp  the  analytic  methods  used,  taking  a  family-of-methods 
approach  that  includes  a  mix  of  lower-  and  higher-resolution  mod¬ 
eling  and  human  gaming  and  integration  thereof.  This  approach 
should  increase  the  emphasis  on  lower-resolution  methods. 

•  Rebalance  the  mix  of  staff  and  use  of  partner  organizations,  such 
as  FFRDCs,  accordingly. 

•  Continuously  invest  in  research  to  support  the  above  analyses 
with  a  broad  range  of  information,  both  qualitative  and  quan¬ 
titative,  and  develop  and  disseminate  new  or  refreshed  methods 
and  tools,  also  both  qualitative  and  quantitative.  The  quality  of 
joint  analysis  depends  on  the  methods  evolving  and  adapting  over 
time. 

These  recommendations  come  with  an  important  caveat.  Half¬ 
way  measures  in  implementation  might  be  costly  and  burdensome 
while  accomplishing  little.  In  particular,  it  will  be  essential  to  change 
cultural  patterns  that  have  hurt  DoD’s  joint  analysis.  These  patterns 
include  a  counterproductive  focus  on  standard  cases  rather  than  plan¬ 
ning  for  broader  capabilities;  a  tendency  to  uncritically  extrapolate  cur¬ 
rent  force  structures  in  PPBE-related  analysis;  a  failure  to  exploit  new 
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technology  and  concepts  of  operation  and  organization;  and  excessive 
dependence  on  large,  complex  models  ill-suited  for  planning  under 
uncertainty. 


APPENDIX  A 


Congressional  Request 


Section  1053  of  the  2015  defense  authorization  bill  called  for  a  study 
with  the  following  elements  (U.S.  Congress,  2014): 

(a)  INDEPENDENT  ASSESSMENT.  The  Secretary  of  Defense 
shall  commission  an  appropriate  entity  outside  the  Department 
Defense  to  conduct  an  independent  assessment  of  the  joint  analytic 
capabilities  of  the  Department  of  Defense  to  support  strategy,  plans, 
and  force  development  and  their  link  to  resource  decisions. 

(b)  ELEMENTS.  The  assessment  required  by  subsection  (a)  shall 
include  each  of  the  following: 

(1)  An  assessment  of  the  analytical  capability  of  the  Office  of  the 
Secretary  of  Defense  and  the  Joint  Staff  to  support  force  planning, 
defense  strategy  development,  program  and  budget  decisions,  and  the 
review  of  war  plans.1 

(2)  Recommendations  on  improvements  to  such  capability  as 
required,  including  changes  to  processes  or  organizations  that  may  be 
necessary. 

(c)  REPORT.  Not  later  than  one  year  after  the  date  of  the  enact¬ 
ment  of  this  Act,  the  entity  that  conducts  the  assessment  required  by 
subsection  (a)  shall  provide  to  the  Secretary  an  unclassified  report,  with 
a  classified  annex  (if  appropriate),  containing  its  findings  as  a  result  of 
the  assessment.  Not  later  than  90  days  after  the  date  of  the  receipt  of 
the  report,  the  Secretary  shall  transmit  the  report  to  the  congressional 
defense  committees,  together  with  such  comments  on  the  report  as  the 
Secretary  considers  appropriate. 
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Support  for  Strategic  Analysis 


The  2001  QDR,  issued  shortly  after  the  September  11  attack  by  al- 
Qaeda  on  New  York  and  the  Pentagon,  has  strong  language  dictating 
changes  in  defense  planning: 

The  approach  shifts  the  focus  of  U.S.  force  planning  from  opti¬ 
mizing  for  conflicts  in  two  particular  regions — Northeast  and 
Southwest  Asia — to  building  a  portfolio  of  capabilities  that  is 
robust  across  the  spectrum  of  possible  force  requirements,  both  func¬ 
tional  and  geographical.  This  approach  to  planning  responds  to 
the  capabilities-based  strategy  outlined  above.  It  focused  more  on 
how  an  adversary  might  fight  than  on  who  the  adversary  might 
be  and  where  a  war  might  occur.  The  shift  is  intended  to  refo¬ 
cus  planners  on  the  growing  range  of  capabilities  that  adversaries 
might  possess  or  could  develop.  It  will  require  planners  to  define 
the  military  objectives  associated  with  defeating  aggression  or 
coercion  in  a  variety  of  potential  scenarios  in  addition  to  con¬ 
ventional  cross-border  invasions.  It  calls  for  identifying,  develop¬ 
ing,  and  fielding  capabilities  that,  for  a  given  level  of  forces,  would 
accomplish  each  mission  at  an  acceptable  level  of  risk  as  established 
by  the  National  Command  Authorities.  (Rumsfeld,  2001,  p.  17; 
emphasis  added) 
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The  Analytic  Agenda 

In  one  of  the  follow-up  actions  to  the  QDR  and  the  related  Defense 
Planning  Guidance,  DoD  issued  a  directive  creating  an  activity,  titled 
Data  Collection,  Development,  and  Management  in  Support  of  Strategic 
Analysis  (DoD  Directive  8260.1,  2002).  This  came  to  be  called  the  ana¬ 
lytic  agenda.  The  purpose  was  to  establish  policy  and  assign  responsibil¬ 
ity  for  “generating,  collecting,  developing,  maintaining,  and  dissemi¬ 
nating  data  on  current  and  future  U.S.  and  non-U.  S.  forces  in  support 
of  strategic  analysis  conducted  by  the  Department  of  Defense.” 

Anticipating  that  a  number  of  planning  scenarios  would  be 
used,  the  directive  called  for  the  construction  of  analytical  baselines 
for  each  scenario.  Such  an  analytic  baseline  would  include  a  scenario, 
CONOPS,  and  integrated  data  to  be  used  as  a  “foundation  for  strategic 
analyses”  in  computer-assisted  wargames  and  theater  campaign  simu¬ 
lations  (DoD  Directive  8260.1,  2002). 

A  scenario,  in  this  context,  meant  a  synopsis  of  the  road  to  war 
and  a  projected  course  of  action  at  the  strategic  and  operational  levels 
of  warfare.  The  synopsis  would  include  political-military  contexts, 
assumptions,  operational  objectives,  major  force  arrivals,  and  plan¬ 
ning  considerations.  Strategic  analysis  would  then  be  accomplished  to 
assess  the  ability  of  current  or  planned  forces  to  address  those  scenarios 
given  priority — i.e.,  to  “execute  the  defense  strategy”  (DoD  Directive 
8260.1,  2002). 

The  primary  responsibilities  were  identified  as  follows: 

•  OUSDP:  develop  and  establish  priorities  among  scenarios  set  in 
future  periods  and  build  scenario  descriptions,  including  road- 
to-war  and  planning  factors  (e.g.,  warning  time,  concurrency, 
assumed  postures  of  engagement) 

•  Chairman  of  the  Joint  Chiefs  of  Staff  (in  coordination  with  ser¬ 
vices):  develop  baselines  for  use  of  current  forces 

•  Director  of  PA&E  (in  coordination  with  the  services):  develop 
baselines  of future  forces,  threats,  and  scenarios;  establish  a  man¬ 
agement  structure  and  data  repository 
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•  service  heads:  support  OSD  and  Joint  Staff  activities  with  data 
and  advice 

•  combatant  commanders:  provide  operational  advice  when  the 
scenario  was  in  their  areas  of  responsibilities  and  generate  current- 
year  analytical  baselines  using  operational  plans 

•  intelligence  community:  provide  projections  of  adversary  future 
force  capabilities  and  capacities. 

Regrettably,  in  light  of  the  intentions  of  the  secretary,  the  directive  said 
nothing  explicit  about  uncertainty.  Although  the  term  baseline  meant 
the  base  from  which  analysis  would  proceed,  the  imperative  to  consider 
extensive  excursions  was  not  mentioned  explicitly.1 

The  subsequent  directive  (DoD  Directive  8260.2,  2003)  was 
about  details  of  implementation  but  also  included  some  clarifications 
and  modifications.  For  example,  the  directive  specified  that  the  Chair¬ 
man  of  the  Joint  Chiefs  of  Staff  would,  in  coordination,  with  the  ser¬ 
vices  and  with  the  director  of  PA&E,  “prepare  annually  an  integrated 
multiyear  program  for  developing  analytic  baselines  for  use  in  strategic 
analyses,  based  upon  scenario  priorities  identified  by  the  [Under  Secre¬ 
tary  of  Defense  for  Policy].”  Thus,  the  role  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff  would  not  be  limited  merely  to  considering  CONOPS 
for  current  forces. 

Again,  the  directive  said  nothing  to  encourage  uncertainty  analy¬ 
sis  except  in  referring  to  analytical  baselines  as  “starting  points”  for 
analysis,  which  might  have  been  read  to  imply  extensive  excursions  but 
was  largely  read  to  mean  starting  points  and  end  points. 

Early  Description  of  Analytic  Agenda  Intentions 

A  rather  detailed  account  of  intentions  for  the  analytic  agenda,  which 
became  the  SSA  activity  after  2010,  was  provided  in  2004  by  CAPE’s 
Deputy  for  Force  Structure  and  Risk  Analysis: 

A  new  DoD  initiative — capabilities  based  planning — shifts  plan¬ 
ning  focus  from  the  performance  of  systems  across  a  relatively 
narrow  range  of  threats  to  the  achievement  of  specific  objectives 
in  response  to  a  broad  set  of  robust  and  adaptive  threats.  It  applies 
to  both  current  and  future  years,  and  makes  trade-offs  across 
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alternative  capabilities  to  accomplish  objectives.  Capabilities- 
based  planning  will  help  frame  the  debate  for  allocating  scarce 
resources  among  diverse  sets  of  alternatives.  It  encompasses  most 
DoD  planning  activities,  including: 

.  .  .  Future  Force  Planning.  Helps  senior  decision  makers  develop 
defense  programs  that  provide  robust  capabilities,  while  minimiz¬ 
ing  risk  within  a  constrained  resource  environment.  .  .  . 

[Sjeveral  activities  .  .  .  will  improve  our  ability  to  perform  analysis 
and  hence  support  capabilities-based  planning.  One  is  the  year- 
old  DoD-wide  initiative  called  the  “analytic  agenda.”  Its  purpose 
is  to  improve  the  quality  and  timeliness  of  analysis,  with  a  scope 
ranging  from  more  comprehensive  scenarios  to  better  data  and 
analytical  tools.  The  end  result  will  be  more  consistent  and  vis¬ 
ible  data  and  analyses.  The  components  of  the  analytic  agenda — 
including  Defense  Planning  Scenarios  (DPSs),  Multi-Service 
Force  Deployment  (MSFD)  data  sets,  and  Analytical  Baselines 
(ABs)  for  analyses  using  future  forces,  along  with  operations 
plans  and  ABs  for  analyses  using  current-year  forces — are  dis¬ 
cussed  below.2 

The  underpinning  of  the  analytic  agenda  is  the  establishment  of 
a  series  of  DPSs.  These  scenarios  represent  a  set  of  realistic  mid¬ 
term  and  far-term  challenges  that  could  require  a  US  military 
response.  DPSs  differ  from  the  Illustrative  Planning  Scenarios 
of  the  past  in  that  they  explicitly  acknowledge  uncertainties  sur¬ 
rounding  our  ability  to  precisely  describe  our  future  adversaries. 

The  MSFD  provides,  for  each  DPS,  order-of-battle  detail  for 
US,  allied,  and  opposing  forces.  It  includes  the  initial  location 
of  units,  the  types  of  equipment  they  possess,  and  how  they  plan 
to  operate.  MSFD  development  is  led  by  the  Joint  Staff,  with  the 
involvement  of  the  intelligence  community,  the  military  services, 
and  the  combatant  commanders. 

A  completed  MSFD  establishes  the  initial  conditions  for  a  postu¬ 
lated  conflict.  An  example  would  be  the  initial  unit  locations  and 
plans  for  a  future  Iraqi  Freedom-like  operation;  another  would  be 
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similar  information  for  a  smaller-scale  contingency,  such  as  our 
operations  in  Bosnia.  We  then  run  this  data  through  our  analytic 
tools  to  forecast  the  results  of  the  conflict.  We  carefully  review 
all  aspects  of  the  conflict — including  deployments,  logistics  and 
engagements — to  ensure  that  the  data  and  models  are  produc¬ 
ing  defensible  results.  After  we  are  satisfied  with  the  analysis, 
we  package  the  associated  data  (DPS,  MSFD,  and  model  with 
inputs  and  outputs)  into  a  product  called  an  Analytical  Baseline 
(AB).  ABs  provide  consistent  and  visible  starting  points  for  future 
analyses.  They  also  greatly  increase  our  ability  to  generate  timely 
answers. 

A  similar  process,  using  combatant  commanders’  operational  plans 
in  place  of  the  DPS  and  MSFD,  develops  ABs  for  current-year  sce¬ 
narios.  Over  the  next  several  years,  this  process  will  establish  five 
to  seven  current-year  ABs,  and  an  equal  number  of  future -year 
ABs  as  starting  points  for  analysis.  In  addition,  for  some  scenarios 
(e.g.,  some  small  scale  contingencies  and  homeland  defense  mili¬ 
tary  actions)  only  the  DPS  and  MSFD  will  be  needed.  No  matter 
what  scenario  or  type  of  operation  is  being  explored,  a  valuable  by¬ 
product  of  the  analytic  agenda  process  is  a  common  language  for 
discussing  and  comparing  analyses.  At  full  maturity,  it  is  antici¬ 
pated  that  most  analyses  reaching  senior  leaders  will  have  been  shared 
and  vetted  through  the  analytic  agenda  process.  Ultimately,  we  expect 
these  common  baselines  to  permit  analytical  excursions  that  will  unveil 
a  wide  range  of  new  opportunities  to  defense  planners. 

To  implement  this  process,  the  Department  issued  guidance 
(DoD  Directive  8260.1  and  DoD  Instruction  8260.2)  establish¬ 
ing  a  common  data-sharing  environment.  The  Joint  Data  Support 
(JDS)  office  has  been  chartered  as  the  single  repository  for  all 
analytical  agenda  data.  (Coulter,  2004;  emphasis  added) 

An  Early  Failed  Attempt  at  a  Strong  Interpretation 

As  discussed  during  my  interviewing  for  this  project,  in  2002-2003, 

OUSDP  attempted  to  mandate  an  interpretation  in  which  SSA  work 

would  systematically  explore  capabilities  for  numerous  versions  of 

each  type  of  scenario  (e.g.,  many  versions  of  a  possible  conflict  with  a 
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regional  power  or  of  a  conflict  with  a  near-peer  power).  The  desire  was 
to  generate  information  about  the  significance  of  such  crucial  assump¬ 
tions  as  strategic  warning,  separation  in  time  between  conflicts,  and 
the  adversary’s  objectives  and  strategy.  The  initiative  was  abandoned 
because  those  responsible  for  joint  analysis  insisted  that  it  was  not  feasi¬ 
ble  to  do  such  exploration.  The  fundamental  problem  was  that  the  Joint 
Staff’s  development  of  the  Multi-Service  Force  Deployment  required 
manpower-intensive  efforts,  by  operations-savvy  officers,  that  sought  to 
develop  realistic  and  responsive  CONOPS.  Doing  so  for  each  scenario 
variant  was  deemed  implausible.  Instead,  the  pattern  became  one  of 
the  Joint  Staff  building  a  well-conceived  CONOPS  for  a  single  version 
of  the  scenario,  albeit  with  a  dutiful  listing  of  the  many  correspond¬ 
ing  assumptions.  In  effect,  the  so-called  analyst  community  (essen¬ 
tially  those  associated  most  closely  with  SSA)  claimed  an  inability  to 
accomplish  the  kind  of  analysis  sought  by  leadership.  This  community 
continued  to  do  analysis  much  as  it  had  in  the  1990s,  using  mostly  the 
same  models  and  methods.  A  major  difference  was  that  everyone  was 
able  to  use  the  same  joint  data  (not  easily  possible  in  the  1990s). 

Viewgraph-level  discussion  obscured  some  of  the  problems.  After 
all,  DoD  was  now  considering  an  increasingly  large  number  of  type 
scenarios  and  thereby  taking  a  much  broader  view  than  it  had  until  the 
mid-1990s.  Indeed,  it  sometimes  seemed  that  DoD  was  considering 
more  scenarios  than  the  system  could  deal  with  well.  The  conceptual 
mischief  here  was  that  the  scenario  used  for  a  given  type  of  conflict 
(e.g.,  major  theater  war  with  a  regional  power)  was  only  one  of  an  infi¬ 
nite  variety.  Although  some  might  imagine  that  this  would  suffice  if 
the  scenario  represented  a  worst  plausible  case,  no  such  single  scenario 
exists  because  a  scenario  posing  a  serious  challenge  in  some  respects 
(e.g.,  short  warning)  necessarily  poses  only  a  modest  challenge  in  other 
respects  (if  the  United  States  has  short  warning,  then  the  adversary 
is  unlikely  to  have  had  substantial  time  for  mobilization).  As  another 
example,  an  adversary  that  is  threatening  and  seriously  intending  to 
use  nuclear  weapons  to  reestablish  deterrence  if  the  United  States  inter¬ 
venes  poses  different  military  problems  than  one  pursuing  a  purely 
conventional  strategy  that  might  eventually  escalate.  As  a  third  exam¬ 
ple,  suppose  that  certain  U.S.  capabilities  are  dramatically  greater  than 
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nominally  assumed  or,  to  the  contrary,  that  certain  adversary  capabili¬ 
ties  are  dramatically  better  than  in  the  current  best  estimate  (e.g.,  vul¬ 
nerabilities  projected  for  an  integrated  air-defense  system  vanish).  To 
understand  the  capabilities  needed  for  any  such  single  type  scenario, 
it  is  necessary  to  consider  a  wide  range  of  variants,  something  recog¬ 
nized  since  the  1980s  (Davis,  1994,  2002).  Doing  so  requires  different 
models  and  analytic  methods  than  those  used  in  the  activities  of  the 
analytic  agenda  and  SSA  (see  the  discussion  in  Chapter  Four).  While 
SSA’s  models  could  and  were  used  for  numerous  excursions  by  CAPE’s 
SAC,  those  were  primarily  excursions  that  were  incremental  in  nature, 
rather  than  ones  posing  distinctly  different  challenges  and,  presum¬ 
ably,  different  operational  strategies  and  related  CONOPS. 


SSA  2011 

A  new  directive  emerged  in  July  2011  (DoD  Directive  8260.05,  2011). 
This  specified  that 

•  SSA  products  shall  include  (1)  current  baselines  reflecting  com¬ 
batant  commander  plans  and  approved  force-management  deci¬ 
sions,  and  (2)  near-  to  long-term  scenarios,  CONOPS,  forces,  and 
baselines  based  on  plausible  challenges  requiring  DoD  resources 
and  capabilities 

•  product  development  shall  be  a  collaborative  and  iterative  process 
led  by  CAPE,  OUSDP,  and  the  Joint  Staff 

•  unresolved  issues  shall  be  referred  to  the  Secretary  of  Defense. 

An  enclosure  to  the  directive  provided  somewhat  more  detail.  For  exam¬ 
ple,  scenario  development  is  referred  to  as  a  responsibility  of  OUSDP, 
but  in  collaboration  with  the  director  of  CAPE  and  the  Chairman  of 
the  Joint  Chiefs  of  Staff  and  in  coordination  with  the  heads  of  OSD 
and  other  DoD  components.  The  Chairman  of  the  Joint  Chiefs  of 
Staff  will  “manage  the  development  of  and  approve  SSA  CONOPS 
and  forces.” 

Figures  B.l  and  B.2  show  SSA  membership  and  the  pre-2012  SSA 
process,  respectively. 
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Figure  B.1 

Participants  in  Support  for  Strategic  Analysis 


DoD  Analysis  Community 
(planning,  programming,  acquisition) 
Guided  by  DODI  8260.01 


SOURCE:  Stevens,  2008.  Later  versions  of  this  briefing  noted  inclusion  of  the 
interagency  community,  industry  and  FFRDCs,  and  allies. 

NOTE:  CAA  =  Center  for  Army  Analysis;  COCOM  =  combatant  commander; 

DIA  =  Defense  Intelligence  Agency;  DISA  =  Defense  Information  Systems  Agency; 
DODI  =  Department  of  Defense  instruction;  DTRA  =  Defense  Threat  Reduction 
Agency;  MCCDC  =  Marine  Corps  Combat  Development  Command; 

NSA  =  National  Security  Agency;  Nil  =  Networks  and  Information  Integration; 
P&R  =  Personnel  and  Readiness. 
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Figure  B.2 

A  Depiction  of  the  Support  for  Strategic  Analysis  Process  (as  of  2011) 


UNCLASSIFIED 


SSA  Process  Life-Cycle 

Spiral  Development 


Stand  alone  scenarios  are  placed  within  the  context  of  Integrated  Security  Constructs 
Spirals  increase  levels  of  detail  as  needed 


Joint  Collaborative  Development 
Scenario:  OSD  (P)  lead 

I 

CONOPS/Forces:  JS  lead 

l 

Analysis:  CAPE  lead 


Spiral  1  (Summary  View) 


1 


SSA  Products 
Scenarios 
CONOPS/Forces 
Baselines 


Spiral  2  (Macro  View) 

L 

Spiral  3  (Detailed  View)  | 

■  SSA  products  support  planning,  programming,  budgeting,  and 
execution  activities  across  the  DoD  (including  acquisition,  concept 
development,  experimentation,  training  and  testing) 


SOURCE:  Excerpted  from  Keener,  2011. 
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APPENDIX  C 


The  Varied  Backgrounds  of  Senior  Department  of 
Defense  Analysts 


In  the  course  of  this  research,  I  was  asked  about  the  disciplinary  back¬ 
grounds  of  notable  DoD  analysts  over  the  years.  Tables  C.l  and  C.2 
give  two  listings.  Table  C.l  shows  the  backgrounds  of  those  who  have 
led  CAPE  or  its  predecessor  organizations.  Table  C.2  shows  recipients 
of  the  most  prestigious  award  of  the  Military  Operations  Research 
Society,  the  Vincent  Wanner  Award.  The  primary  observation  is  that 
the  range  of  backgrounds  is  substantial  and,  in  particular,  is  not  par¬ 
ticularly  associated  with  applied  mathematics  or  operations  research. 

For  some  further  data,  Table  C.3  shows  the  range  of  backgrounds 
of  RAND  staff  with  doctorates.  Roughly  half  or  more  of  RAND  staff 
work  on  non-DoD  problems,  but  the  same  range  (if  not  the  percent¬ 
ages)  exists  for  national-security  work  (see  Figure  C.l).  Figure  C.2  is  a 
roughly  comparable  chart  for  the  Institute  for  Defense  Analyses,  which 
is  much  more  heavily  dedicated  to  national-security  work.1  As  a  final 
data  point,  the  CAPE  website  states  that 

CAPE’s  civilian  and  military  staff  is  highly  educated,  with  back¬ 
grounds  in  a  variety  of  academic  disciplines,  including  physics, 
economics,  engineering,  mathematics,  biology,  computer  science, 
and  operations  research.  Approximately  35%  of  the  analytic  civil¬ 
ian  staff  holds  a  doctorate  in  their  field,  and  a  substantial  share 
have  prior  military  experience.  (CAPE,  undated) 
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Table  C.1 

Disciplinary  Backgrounds  of  CAPE  Leadership 


Name 

Discipline 

Date 

Alain  Enthoven 

Economics 

1965-1969 

Ivan  Selin 

Engineering 

1969-1970 

Gardiner  Tucker 

Physics 

1970-1973 

Leonard  Sullivan 

Engineering 

1973-1976 

Edward  C.  Aldridge 

Engineering,  science 

1976-1977 

Russell  Murray 

Engineering 

1977-1981 

David  C.  Chu 

Economics 

1981-1993 

William  J.  Lynn 

Law,  public  affairs 

1993-1997 

Robert  R.  Soule 

Economics 

1998-2001 

Barry  D.  Watts 

History 

2001-2002 

Stephen  A.  Cambone 

Political  science 

2002-2003 

Kenneth  J.  Krieg 

Public  policy 

2003-2005 

Bradley  M.  Berkson 

Engineering,  business  administration 

2005-2009 

Christine  H.  Fox 

Mathematics 

2009-2014 

Jamie  M.  Morin 

Public  policy  and  administration, 
political  science,  engineering 

2014-present 

NOTE:  At  different  points. 

the  office  has  been  called  Systems  Analysis, 

Program 

Analysis  and  Evaluation,  and  Cost  Assessment  and  Program  Evaluation. 


Other  contributors  to  SSA  (such  as  those  in  OUSDP)  typically  have 
backgrounds  in  political  science,  international  relations,  and  other 
social  science  subjects. 
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Table  C.2 

Winners  of  the  Vincent  Wanner  Award 


Name 

Disciplinary  Background 

Date 

Mr.  Ervin  Kapos 

Mathematics 

2015 

Mr.  Jack  F.  Keane 

Operations  research 

2014 

Dr.  Gregory  S.  Parnell 

Engineering-economic  systems 

2013 

Brigadier  General  (Retired)  Michael  L. 
McGinnis 

Systems  and  industrial  engineering 

2012 

Dr.  Andrew  G.  Loerch 

Engineering  and  operations 
research 

2011 

Mr.  Andrew  W.  Marshall 

Economics 

2010 

Dr.  Stuart  H.  Starr 

Electrical  engineering 

2009 

Dr.  Thomas  L.  Allen 

Economics  and  operations  research 

2008 

Mr.  Walter  F.  Hollis 

Mathematics,  physics 

2007 

Dr.  Roy  E.  Rice 

Operations  research 

2006 

Dr.  Vernon  Bettencourt 

Operations  research 

2005 

General  David  M.  Maddox 

Operations  research 

2004 

Ms.  Natalie  W.  Crawford 

Mathematics 

2003 

Mr.  Michael  F.  Bauman 

Engineering 

2002 

Dr.  William  G.  Lese 

Mathematical  statistics,  computer 
science 

2001 

Mr.  Seymour  J.  Deitchman 

Engineering 

2000 

Dr.  Donald  B.  Rice 

Engineering,  economics 

1999 

General  Larry  D.  Welch 

Business  administration, 
international  relations 

1998 

Dr.  Paul  K.  Davis 

Chemical  physics 

1997 

Dr.  Edward  C.  Brady 

Electrical  engineering, 
mathematical  economics 

1996 

Mr.  E.  B.  Vandiver  III 

Physics 

1995 

Mr.  James  N.  Bexfield 

Mathematics,  operations  research 

1994 

Dr.  David  S.  C.  Chu 

Economics 

1993 
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Table  C.2 — Continued 


Name 

Disciplinary  Background 

Date 

Major  General  John  D.  Robinson 

Business  administration 

1992 

Dr.  Marion  L.  Williams 

Engineering 

1991 

Mr.  John  K.  Walker 

Education 

1990 

Mr.  Wayne  P.  Hughes 

Operations  research,  mathematical 
statistics 

1989 

Clayton  J.  Thomas 

Mathematics 

1988 

Wilbur  B.  Payne 

Physics 

1987 

Dr.  Seth  Bonder 

Industrial  engineering 

1986 

Marion  R.  Bryson 

Mathematics 

1985 

David  A.  Schrady 

Operations  research 

1984 

Major  General  Jasper  A.  Welch,  Jr. 

Physics 

1983 

Mr.  Walter  L.  Deemer 

Mathematics  (imputed) 

1982 

Dr.  Jack  R.  Borsting 

Mathematics 

1981 

Lieutenant  General  Glenn  A.  Kent 

Meteorology,  mathematics,  physics 

1980 

Dr.  Bernard  0.  Koopman 

Mathematics 

1979 

Dr.  Philip  M.  Morse 

Physics 

1978 

NOTES:  The  Vincent  Wanner  Award  is  the  most  prestigious  award  of  the  Military 
Operations  Research  Society.  The  disciplinary  backgrounds  were  obtained  via  web 
searches  and  should  not  be  considered  definitive. 
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Figure  C.1 

Disciplinary  Background  of  RAND  PhD  Staff 


■  Economics,  social  sciences, 
and  political  sciences 

■  Physical  sciences 

■  Computer  science 

■  Mathematics,  operations 
research,  and  statistics 

■  Life  sciences 

■  Engineering 
Business  and  law 


SOURCE:  Calculated  using  data  from  RAND  Corporation,  undated. 
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Figure  C.2 

Disciplinary  Background  of  the  Institute  for  Defense  Analyses  Staff 


■  Economics,  social  sciences, 
and  political  sciences 

■  Physical  sciences 

■  Computer  science 

■  Mathematics,  operations 
research,  and  statistics 

■  Life  sciences 

■  Engineering 
Business  and  law 


SOURCE:  Calculated  using  data  from  Institute  for  Defense  Analyses,  undated. 
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APPENDIX  D 


Options  for  Model  Development,  Maintenance, 
and  Usage 


This  report  calls  for  the  reinvigoration  of  DoD’s  capabilities  for  joint 
modeling  and  analysis,  with  a  mix  of  model  types,  human-gaming, 
and  other  research.  A  question  that  arose  repeatedly  during  the  study 
was  where  the  locus  of  such  work  would  be.  Originally,  the  context  was 
how  to  restore  the  campaign-modeling  that  existed  in  2011,  in  CAPE’s 
SAC,  but  the  scope  is  now  much  broader.  Some  of  the  possibilities  con¬ 
sidered  in  the  study  were 

1.  none  (there  would  be  no  DoD-coordinated  function,  except 
possibly  for  joint  data  collection  and  maintenance) 

2.  in  the  government  (CAPE,  J-8,  a  new  entity  in  National  Defense 
University,  Net  Assessment,  or  other) 

3.  FFRDCs. 

Under  any  of  these,  commercial  contractors  might  be  used  for  some 
developments. 

The  first  option  is  clearly  feasible  and,  to  a  considerable  extent, 
represents  the  status  quo:  Components  award  contracts  as  they  see 
necessary,  whether  to  private  contractors  or  FFRDCs.  Assuming  that 
joint  data  remain  an  SSA  function  (or  a  function  of  the  new  ASSP), 
the  approach  would  probably  continue  to  suffice,  although  with 
inefficiencies. 

Several  options  for  in-government  leads  exist,  as  indicated,  but  it 
seems  unlikely  that  CAPE  or  the  Joint  Staff  will  take  on  the  respon¬ 
sibility  without  significant  increases  in  billets  and  funding,  as  well  as 
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persuasive  arguments  about  the  need  to  do  so.  Creating  a  new  entity 
in  the  National  Defense  University  would  be  a  risky  endeavor  with 
a  long  buildup  time;  Net  Assessment  would  be  a  poor  choice  for  the 
same  reason  and  because  the  office  has  traditionally  had  very  different 
talents  and  niches.1  A  primary  consideration  in  this  judgment  is  that 
developing  the  necessary  analytic  talent  and  organization  is  a  substan¬ 
tial  challenge.  High-quality  modelers  and  analysts  cannot  be  quickly 
found  and  hired,  and  a  high-quality  organization  to  perform  these 
functions  simply  could  not  arise  overnight.  Because  the  in-government 
options  did  not  appear  attractive,  I  gave  more  consideration  to  FFRDC 
options. 


Ad  Hoc  or  with  Coordination  by  ASSP 

For  some  years,  DoD  components  have  used  FFRDCs  on  an  ad  hoc 
basis  for  a  wide  range  of  studies  on  broad  and  focused  strategic  research, 
particular  military  challenges,  and  particular  examples  of  modeling  or 
human  wargaming.  The  default  option  is  simply  to  continue.  Although 
the  term  ad  hoc  can  be  pejorative,  that  is  not  intended  here.  This  option 
would  not  require  any  new  administrative  apparatus.  If  an  ASSP  is 
created,  then  one  of  its  functions  could  be  to  coordinate  arrangements 
with  the  FFRDCs  (with  CAPE  and  the  Joint  Staff  taking  the  lead).  It 
would  be  essential  to  ensure  that  the  FFRDCs  undertaking  the  work 
understood  the  spirit  intended,  as  discussed  throughout  this  report 
in  connection  with  the  ASSP.  Otherwise,  the  result  might  have  an 
FFRDC  label  but  would  have  the  same  shortcomings  as  the  earlier 
campaign-modeling  that  CAPE  chose  to  dissolve  in  2011. 


A  WSEG  Model 

An  alternative  approach  would  be  more  highly  structured.  From  1948 
to  1976,  the  Institute  for  Defense  Analyses  had  a  division  called  the 
Weapon  Systems  Evaluation  Group  (WSEG)  attached  to  the  Joint  Staff 
and  usually  headed  by  a  serving  three-star  military  officer  (Ponturo, 
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1979).  WSEG’s  studies  were  almost  all  for  the  Joint  Chiefs  of  Staff  or 
the  Director  for  Defense  Research  and  Engineering,  who  was  rhe  high¬ 
est  official  under  rhe  secretary  and  deputy  secretary.  The  first  director 
was  Philip  M.  Morse,  who  returned  to  his  MIT  professorship  in  1950. 
The  final  director  was  Lt  Gen  Glenn  Kent  (U.S.  Air  Force).  WSEG 
was  a  wholly  in-house  organization  with  about  50  professionals,  half 
military  and  half  civilian.  WSEG  was  able  to  hire  top-notch  talent 
without  the  then-stringent  restrictions  on  salary.  In  the  early  years, 
WSEG  played  a  preeminent  role  in  DoD  analysis.  That  changed  in  the 
1960s,  with  the  emergence  of  such  offices  as  OSD’s  Systems  Analy¬ 
sis  (later  PA&E  and  CAPE),  other  FFRDCs  (e.g.,  RAND,  Center  for 
Naval  Analyses,  Research  Analysis  Corporation,  Aerospace,  MITRE), 
and  the  greatly  increased  analytic  capabilities  in  the  services  and  the 
Joint  Chiefs  of  Staff.  The  Secretary  of  Defense  disestablished  WSEG  in 
1976,  writing:  “It  is  no  longer  needed  .  .  .  given  the  extensive  complex 
of  study  and  evaluation  activities  available  to  the  Department”  (Pon- 
turo,  1979). 


A  Distributed  WSEG  Model 

An  alternative  approach  modernizing  from  the  WSEG  experience 
might  be  more  suitable  today.  It  would  use  two  or  more  FFRDCs  for 
different  facets  of  support  to  joint  analysis.  With  modern  network¬ 
ing,  the  FFRDCs  could  be  functionally  well  integrated.  FFRDCs  have 
different  relative  strengths  and  different  relative  interests  in  various 
types  of  analysis.  Those  can  change  over  time.  In  practice,  something 
like  this  (using  multiple  FFRDCs)  is  the  situation  today,  but  without 
any  special  organizational  or  legal  relationships.  Different  DoD  com¬ 
ponents  ask  their  FFRDCs  for  particular  studies  or  activities  on  an 
ad  hoc  basis.  Congress  also  mandates  some  studies  that  are  ultimately 
accomplished  by  FFRDCs  (including  this  report).2  The  activities  are 
not,  however,  coordinated,  much  less  integrated. 

A  possible  advantage  to  a  “modern  WSEG,”  at  least  for  niche  pur¬ 
poses,  would  be  that  it  might  make  it  far  easier  to  have  a  stable  group 
of  top-flight  analysts  who  were  read  into  the  various  compartmented 
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programs  on  which  independent  analysis  is  needed.  That  would  require 
significant  early  effort,  but  the  result  might  then  be  maintainable  and 
efficient.  Dealing  with  compartmented  information  has  become  a 
major  problem  over  the  past  decade  or  so,  within  the  DoD  itself  and 
FFRDCs. 


Notes 


Notes  to  Summary 

1  The  words  have  distinct  meanings  here.  They  refer  to  flexibility  across  missions, 
adaptiveness  to  different  circumstances,  and  robustness  to  shocks.  Authors  often  use 
one  of  these  words  as  shorthand  for  all  three,  as  in  exhorting  planning  for  adaptive¬ 
ness  or  seeking  robust  capabilities.  The  2010  Quadrennial  Defense  Review  also  used 
the  word  versatility  in  this  way.  Still  other  authors  use  the  term  agility. 

2  The  elements  of  the  list  relate  to  what  DoD  variously  refers  to  as  challenges,  illus¬ 
trative  planning  scenarios,  integrated  security  constructs,  and  defense-planning  scenarios. 

3  Low  resolution  can  be  illustrated  with  two  examples:  (1)  a  strategic-nuclear  sce¬ 
nario  specifies  the  alert  state  of  U.S.  forces  and  the  type  of  retaliatory  attack  for 
which  it  is  ready;  (2)  a  strategic-mobility  scenario  requires  airlift  capacity  as  mea¬ 
sured  in  millions  of  ton-miles  per  day,  without  specifying  what  is  lifted  or  where  it 
goes.  Such  low-resolution  depictions  should  be  approximately  consistent  with  more- 
detailed  descriptions. 


Notes  to  Chapter  One 

1  This  research  adhered  to  policies  approved  by  RAND  and  DoD  for  the  protec¬ 
tion  of  human  subjects.  Those  DoD  officials  interviewed  provided  their  personal 
views  and  did  not  represent  any  position  of  DoD. 

2  The  materials  included  overview  briefings  of  the  SSA  activity,  completed  ana¬ 
lytic  baselines,  current  memoranda  describing  in-progress  work,  briefings  sent  to 
high  officials,  and  published  papers  by  interviewees.  Those  interviewed  were  chosen 
to  ensure  hearing  from  personnel  in,  or  previously  in,  each  of  the  major  offices 
in  the  Office  of  the  Secretary  of  Defense  (OSD),  the  Joint  Staff,  and  the  services. 
By  the  midpoint  of  interviewing,  very  few  new  issues  and  points  of  disagreement 
were  being  mentioned.  Instead,  interviewees  were  presenting  examples  and  nuances 
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reflecting  their  personal  experiences.  The  review  process  served  as  another  check  to 
ensure  that  the  most  consequential  issues  and  disagreements  had  been  heard. 

3  Campaign  analysis  studies  a  phase  of  war  that  has  a  series  of  operations  to  accom¬ 
plish  military  objectives.  Two  illustrative  campaigns  were  the  invasion  of  Iraq  in 
2003  and  the  surge  in  Iraq  in  2007.  Campaign  analysis  for  hypothetical  scenarios  is 
used  to  evaluate  future  force  structures  and  capability  mixes.  Scenarios  may  involve 
concurrent  crises  across  the  globe.  Campaign  analysis  may  be  accomplished  with 
complex  computer  simulation  models,  simpler  models,  and  human  wargaming. 

4  The  services  retain  these  capabilities  and  federally  funded  research  and  develop¬ 
ment  centers  (FFRDCs)  often  conduct  such  analysis  for  OSD,  the  Joint  Staff,  or 
services.  Some  officers  on  joint  assignments  can  perform  campaign  analysis  even  if 
they  are  not  using  campaign  models. 

3  The  results  of  these  models  can  be  understood  and  explained  by  experts  familiar 
with  the  models,  but  not  necessarily  easily.  Such  explanations  are  beyond  the  abili¬ 
ties  of  those  who  can  run  the  model  but  do  not  appreciate  underlying  details  or  do 
work  at  code  level.  An  expert  analyst,  if  unfamiliar  with  model  details  or  lacking 
the  considerable  time  to  deal  with  subtleties,  may  not  find  the  model’s  behavior  to 
be  intuitive.  Some  interviewees  said,  bluntly,  that  they  “did  not  trust”  the  models  or 
data. 

6  The  shift  to  capability-based  planning  was  made  in  2001  (Rumsfeld,  2001),  pre¬ 
saged  by  the  first  QDR,  which  referred  to  being  able  to  “respond  to  the  full  spec¬ 
trum  of  crises”  (Cohen,  1997).  The  shift  had  been  urged  in  earlier  books  (e.g.,  Davis, 
1994)  and  reports  (e.g.,  National  Defense  Panel,  1997).  As  discussed  elsewhere 
(Davis,  2014,  p.  107),  DoD’s  initial  implementation  had  complex  processes  and  led 
to  serious  misconceptions,  such  as  the  incorrect  notion  that  the  new  approach  did 
not  use  scenarios,  name  specific  threats,  or  deal  with  resource  constraints. 

7  The  need  for  continued  vigorous  change  has  been  recognized  across  adminis¬ 
trations  (Rumsfeld,  2006;  Gates,  2010;  Plagel,  2014) — most  recently  by  Secre¬ 
tary  Ashton  Carter.  The  strategic  issues  have  been  described  in  terms  of  “eroding 
foundations  of  American  power”  (Krepinevich,  2009)  and  “looming  discontinui¬ 
ties”  (Davis  and  Wilson,  2011).  To  say  the  least,  recent  trends  are  not  favorable 
(Ochmanek,  2014). 

8  Early  materials  included  Program  Analysis  and  Evaluation  (1979),  a  review  of 
base  force  development  (Jaffe,  1993),  a  review  of  the  two-decade  period  of  planning 
documents,  starting  in  1980  (Larson,  Orletsky,  and  Leuschner,,  2001),  and  a  recent 
update  (Larson  et  ah,  forthcoming). 

9  These  included  the  Bottom  Up  Review  and  subsequent  QDRs  (Aspin,  1993; 
Cohen,  1997;  Rumsfeld,  2001;  Rumsfeld,  2006;  Gates,  2010;  FFagel,  2014),  as  well 
as  an  implementation  plan  (Joint  Defense  Capabilities  Study  Team,  2004),  Defense 
Science  Board  reports  (Defense  Science  Board,  1996a,  1996b,  1998,  2003a,  2003b, 
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2009),  and  a  study  for  the  Assistant  Secretary  for  Defense  for  Research  and  Engi¬ 
neering  (Mullen,  2013). 

10  Certain  National  Academies  of  Sciences,  Engineering,  and  Medicine  reports 
were  helpful  (National  Research  Council,  1997,  2005,  2006).  Some  FFRDC  stud¬ 
ies  addressed  fundamentals  (Davis,  2014),  and  some  reports  were  about  rethink¬ 
ing  analytic  methods  (Allen  et  al.,  2007;  Davis  and  Henninger,  2007)  or  based  on 
working  directly  with  high  officials  (Davis,  Shaver,  and  Beck,  2008). 

11  Many  informal  materials  are  in  the  archives  of  the  Military  Operations  Research 
Society.  The  most-relevant  materials  described  meetings  before  or  after  QDRs;  these 
demonstrate  efforts  to  take  on  new  challenges  (Bexfield  and  Disbrow,  2004;  Allen 
and  Bexfield,  2006;  Patenaude  and  Stanic,  2010;  Davis,  1998;  Leonard,  Bexfield, 
and  Sharman,  1998;  Leonard  and  Bexfield,  2000;  Thomason  and  Dechant,  2008; 
Solly,  2014;  Military  Operations  Research  Society,  2013;  Leonard,  Thomas,  and 
Bexfield,  2013).  See  also  a  briefing  on  British  work  (Solly,  2015). 

12  See,  e.g.,  a  report  from  the  Technical  Cooperation  Program  (Taylor,  2011). 

13  The  best-known  account,  originally  published  in  1971,  describes  early  systems 
analysis  (Enthoven  and  Smith,  2005),  calling  on  a  decade’s  efforts  for  the  Air 
Force  (Hitch  and  McKean,  1960).  Broader  discussions  appeared  in  edited  volumes 
(Quade,  1966;  Quade  and  Boucher,  1968;  Miser  and  Quade,  1988;  Quade  and 
Carter,  1989).  William  Kaufmann,  an  advisor  to  several  defense  secretaries,  wrote 
about  force-structuring  analysis  (Kaufmann,  1981,  1986,  1991). 
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1  That  SSA  activities  serve  very  different  functions  is  discussed  in  Fitzsimmons, 
2012,  and  Dechant  et  ah,  2008.  One  theme  of  the  latter  is  the  need  for  unclassi¬ 
fied  variants  of  scenarios  that  can  be  used  readily  in  educational  settings.  A  theme 
of  the  former  is  that  scenario  needs  are  different  for  strategy  development  and  pro¬ 
gram  development.  Fitzsimmons  also  argues  for  putting  more  relative  emphasis  on 
analysis  rather  than  scenario  development. 

2  This  draws  on  Davis,  2014,  which  reflects  ideas  and  methods  developed  specifi¬ 
cally  for  two  Under  Secretaries  for  Acquisition,  Technology  and  Logistics  (AT&L). 
The  context  was  capability-area  reviews  (Davis,  Shaver,  and  Beck,  2008),  but  the 
approach  applies  equally  well  to  Secretary  of  Defense  issues  affecting  the  size  and 
capabilities  of  future  forces. 

3  A  senior  leader  will  seldom  have  time  to  delve  into  details  but  may  do  so  selec¬ 
tively — to  test  the  quality  of  staff  work  or  to  better  understand  a  crucial  issue.  Orga¬ 
nizing  decision- aiding  so  that  real-time  zooming  is  possible  sharpens  work  even  if 
zooming  is  invoked  rarely. 


84  Capabilities  for  Joint  Analysis  in  the  Department  of  Defense 


I  In  private  industry,  the  function  is  sometimes  served  by  management  consul¬ 
tants,  such  as  McKinsey  and  Company  or  the  Boston  Consulting  Group,  reporting 
to  the  chief  executive  officer. 

5  The  words  have  distinct  meanings  here  and  refer  to  flexibility  across  missions, 
adaptiveness  to  different  circumstances,  and  robustness  to  shocks.  Authors  often  use 
one  word  as  shorthand  for  all  three,  as  in  exhorting  planning  for  adaptiveness  or 
seeking  robust  capabilities.  The  2010  QDR  used  the  word  versatility  in  this  way.  Still 
other  authors  use  the  term  agility,  as  in  some  of  NATO’s  work  (Alberts,  2011). 

6  See  Davis,  2014.  The  new  responsibility  has  long  been  desired,  as  suggested  in 
a  1992  talk  by  the  director  of  PA&E  exhorting  analysts  to  discuss  why  conclusions 
emerge,  bow  results  would  change  with  different  assumptions,  why  to  believe  that  a 
particular  system  or  capability  is  needed,  why  new  and  varied  scenarios  are  needed, 
and  why  we  need  to  plan  forces  that  will  be  used  in  a  wide  variety  of  future  situations 
(Chu,  1992). 

7  Usage  is  inconsistent,  but  measure  refers  to  an  amount  or  degree  of  something.  A 
metric  is  a  standard  of  measurement,  such  as  the  ratio  between  an  amount  and  an 
amount  regarded  as  adequate. 

8  As  an  example,  circa-2000  guidance  for  air  forces  to  plan  capabilities  for  halting 
an  advancing  mechanized  army  meant  little  without  indicating  how  big  an  army 
could  be  thwarted,  how  quickly,  and  whether  air  supremacy  should  be  assumed. 

9  As  one  example,  the  armored-vehicle  kills  per  day  by  air  forces  in  a  continuing 
campaign  against  an  invader  depend  on  lower-level  component  factors,  such  as  sor¬ 
ties  per  day  and  kills  per  sortie,  while  kills  per  sortie  depend  on  still-lower-level 
factors,  such  as  kills  per  engagement  and  the  number  of  engagements  per  sortie  (a 
function  of  command  and  control).  Improving  the  top-most  measure  of  effective¬ 
ness  might  actually  be  achieved  by  reducing  sorties  per  day  while  increasing  engage¬ 
ments  per  sortie  and  kills  per  engagement.  Merely  flying  more  sorties  per  day  might 
be  counterproductive  if  it  came  at  the  expense  of  per-engagement  effectiveness. 

10  The  Secretary  of  Defense  issues  contingency  planning  guidance,  approved  by 
the  President,  for  the  Chairman  of  the  Joint  Chiefs  of  Staff  to  use  in  developing  the 
Joint  Strategic  Capabilities  Plan.  OUSDP  has  the  lead  role  in  preparing  the  contin¬ 
gency  planning  guidance  and  reviewing  subsequent  plans.  The  legislation  is  United 
States  Code,  Title  10,  Section  113  (g)(2). 

II  The  battle  for  resources  is  fundamental  for  officers  working  in  the  Pentagon,  as 
recounted  in  a  classic  book  (Smith  and  Gerstein,  2007). 


Notes  to  Chapter  Three  85 


Notes  to  Chapter  Three 

1  Campaign  analysis  can  be  accomplished  without  using  the  particular  campaign 
models  of  earlier  SSA,  or  indeed  without  any  computer  models,  but  that  has  been  rare 
within  DoD.  Many  of  those  interviewed  for  this  study  decried  the  reduced  capabil¬ 
ity  in  OSD  and  the  Joint  Staff  for  campaign  analysis,  not  just  campaign-modeling. 

2  Some  of  those  critical  of  SSA  cited  anecdotes  in  which  SSA  representatives 
based  assertions  on  questionable  assumptions,  but  the  vast  majority  of  those  inter¬ 
viewed  agreed  that  SSA’s  analytic  baseline  work  was  well  conceived  and  managed.  It 
included  innovative  efforts  to  use  human  wargaming  and,  in  recent  years,  to  apply 
social-science  experts. 

3  This  judgment  reflects  a  RAND  study  for  the  U.S.  Army,  which  concluded  that 
SSA  (or,  more  precisely,  the  efforts  of  the  earlier  analytic  agenda  activity)  had  proven 
markedly  valuable  for  preparation  of  the  2010  QDR,  much  more  so  than  for  the 
2006  QDR  (Larson  et  al.,  forthcoming). 

4  More  precisely,  the  senior  analysis  officials  associated  with  SSA  supported  research 
funded  through  the  Modeling  and  Simulation  Coordination  Office  (MSCO).  Pub¬ 
lished  examples  include  Davis  and  Cragin,  2009;  Davis,  2011;  and  a  variety  of  clas¬ 
sified  reports.  For  an  overview  see  Coulter,  2009.  For  discussion  in  terms  of  unre¬ 
stricted  warfare,  see  Akst,  2009.  MSCO  also  sponsored  and  participated  in  related 
special  meetings  of  the  Military  Operations  Research  Society.  Such  efforts  continue, 
as  illustrated  in  2016  by  special  meetings  on  human  wargaming. 

5  This  is  based  on  my  discussions  in  the  mid-1990s  with  the  OSD  officials  Ted 
Warner  (then  Assistant  Secretary  of  Defense  [ASD]  for  Strategy  and  Resources)  and 
Fred  Frostic  (in  Warner’s  office),  as  well  as  more-recent  discussions  with  the  office. 

6  PPBE-related  analysts  tend  to  worry  about  cost-effectiveness  and  to  be  skepti¬ 
cal  about  how  quickly  new  capabilities  will  emerge  and  how  effective  they  will  be. 
Analysts  associated  with  research  and  development  tend  to  be  more  technology- 
push  oriented  and  to  be  optimistic  about  the  effectiveness  of  future  capabilities  and 
the  speed  with  which  they  could  be  attained.  Both  sides  can  point  to  supportive 
examples.  Precision  fires  and  precision  navigation  transformed  U.S.  military  opera¬ 
tions  in  the  course  of  a  decade  but  proved  far  less  effective  when  Serbia’s  tanks  hid 
within  forests  and  when  manpower-intensive  operations  were  necessary  in  counter¬ 
insurgency.  DoD  needs  healthy  competition  between  the  two  views. 

7  Some  compartmented  programs  are  aggressive  and  forward-looking.  Some  in 
SSA  leadership  have  the  necessary  clearances.  It  is  often  possible  to  conduct  related 
analysis  using  modifications  of  mainstream  SSA  data  and  models.  Reportedly,  how¬ 
ever,  there  are  limitations  in  the  ability  to  do  fully  joint  work,  as  noted  by  a  long¬ 
time  SSA  leader  and  chief  Navy  analyst,  Trip  Barber  (Barber,  2015). 
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8  This  later  became  the  Modeling  and  Simulation  Coordination  Office. 

9  The  Multi-Service  Force  Deployments  assume  adversary  characteristics  based 
on  intelligence  projections.  In  one  account,  a  Multi-Service  Force  Deployment  can 
be  a  500-page  description  of  how  the  postulated  conflict  unfolds.  The  description 
includes  data  for  orders  of  battle,  strategy  and  tactics  at  the  operational  level  of  war, 
axes  of  attack,  defensive  dispositions,  tables  of  equipment,  force  allocations  to  mis¬ 
sions,  operations  tempo  and  sortie  rates,  readiness  factors,  munitions,  and  sustain¬ 
ment  (Cerniglia-Mosher,  2009). 

10  The  background  research  includes  in-house  work  and  sponsored  research  at 
FFRDCs  (e.g.,  Tostumbo  et  al.,  2013,  2016).  Some  of  this  has  also  helped  to  develop 
the  defense  planning  scenarios. 

11  The  models  in  question,  known  by  their  acronyms,  have  included  TACWAR, 
TFiUNDER,  JICM,  and  STORM.  All  are  good  models  but  complex.  JICM  was 
originally  designed  to  be  more  strategic  level  in  nature  and  be  correspondingly  sim¬ 
pler,  but  embellishments  and  practice  have  made  it  less  so. 

12  SSA  recognized  this  problem  and  supported  some  exceptional  efforts.  In  particu¬ 
lar,  the  Marine  Corps  Combat  Development  Center  conducted  an  extensive  study, 
the  Joint  Irregular  Warfare  Analytic  Baseline  (JIWAB)  (Wong,  2015),  that  broadly 
approached  a  challenging  intervention  scenario  with  interagency  representatives  and 
social-science  experts  with  varied  conceptual  frameworks  and  tools.  The  intent  was 
to  think  through  a  CONOPS  that  would  properly  deal  with  the  political,  economic, 
and  human  issues  often  omitted  in  military  analysis.  The  study  was  never  formally 
turned  into  an  analytic  baseline,  but  it  illustrated  what  the  analytic  community 
might  do  if  unshackled  from  traditional  methods  and  models.  Other  SSA  efforts 
that  did  not  involve  warfighting  campaign  models  included  use  of  the  British  Peace 
Support  Operations  Model  (PSOM)  (Body  and  Marston,  2011)  and  looking  at  the 
problem  of  securing  nuclear  weapons  in  a  chaotic  environment. 

13  This  point  was  made  in  the  2010  QDR  (Gates,  2010),  the  development  of  which 
had  included  a  good  deal  of  scenario  variation  in  much  the  same  spirit  as  described 
here.  That  QDR  made  some  use  of  SSA  products  (data  and  baselines)  but  was  nota¬ 
ble  for  the  low-resolution  analysis  that  addressed  other  issues  and  different  combina¬ 
tions  of  stress  on  U.S.  military  capabilities.  Significantly,  that  analysis  depended  on 
expertise  available  in  CAPE’s  Simulation  and  Analysis  Center  because  of  the  prior 
in-depth  campaign  analysis,  even  though  the  analysis  made  little  direct  use  of  the 
campaign  models. 

14  In  this  report,  capability-based  planning  means  “[pjlanning,  under  uncertainty, 
to  provide  capabilities  suitable  for  a  wide  range  of  modern-day  challenges  and  cir¬ 
cumstances  while  working  within  an  economic  framework  that  necessitates  choice” 
(Davis,  2002,  p.  xi).  In  this  context,  capability  covers  what  others  refer  to  as  capabili¬ 
ties  and  capacities.  The  term  capability-based  planningh&s  baggage  because  of  DoD’s 
notoriously  complex  processes  for  its  implementation  and  basic  misunderstandings, 
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including  a  false  contrast  with  threat-based  planning.  Capability-based  planning 
makes  extensive  use  of  scenarios  and  considers  both  generic  and  specific  threats.  The 
confused  history  is  described  elsewhere  (Davis,  2014). 

15  This  can  be  seen,  for  example,  in  congressional  testimony  by  a  recent  defense 
official  about  U.S.  challenges  in  the  western  Pacific,  which  discusses  numerous 
problems  and  the  importance  of  the  “third  offset  strategy”  announced  by  Secretary 
Chuck  Hagel  (Ochmanek,  2014). 

16  Standard  academic  references  include  the  work  of  Henry  Mintzberg  (Mintzberg, 
1994;  Mintzberg,  Lampel,  and  Ahlstrand,  2005). 

17  Excellent  use  of  optimization  can  be  found  in  numerous  past  studies,  by  both 
DoD  and  a  few  foreign  militaries,  such  as  the  British  Defence  Operational  Analysis 
Establishment  in  supporting  a  major  UK  review  (Hoehl  and  Scales,  2011). 

18  Various  serving  or  past  officials  have  lamented  the  consensus  problem  while 
making  other  suggestions  regarding  how  to  improve  strategic  planning  in  DoD 
(Flournoy,  2015;  Hicks,  2014). 


Notes  to  Chapter  Four 

1  One  version  was  presented  as  early  as  2004  by  Kenneth  Krieg,  then  director  of 
PA&E  and  later  USD(AT&L)  (Bexfield  and  Disbrow,  2004). 

2  Because  of  resource  limitations,  this  study  did  not  address  current  and  near- 
term  planning,  which  use  similar  methods  but  focus  on  current  war  plans  and  their 
extrapolation. 

3  The  elements  of  the  list  relate  to  what  DoD  variously  refers  to  as  challenges,  illus¬ 
trative  planning  scenarios,  integrated  security  constructs,  and  defense-planning  scenarios. 

4  Historical  examples  include  the  introduction  of  precision  fires  and  network¬ 
ing,  armed  aerial  combat  vehicles,  and  the  scale-up  of  such  unmanned  operations. 
Looking  forward,  some  examples  might  be  found  in  the  systems  being  examined  in 
DoD’s  Strategic  Capabilities  Office  (Carter,  2016).  These  reportedly  include  use  of 
microdrone  packs  for  surveillance  and  other  functions,  new  concepts  for  infantry, 
new  roles  for  unmanned  fighters  (the  Loyal  Wingman  initiative),  and  arsenal  air¬ 
craft  (Lamothe,  2016). 

5  Some  argue  that  having  the  Joint  Staff  take  on  this  role  is  implausible,  but  it 
should  be  noted  that  during  the  1990s,  the  Joint  Staff  was  an  important  proponent 
of  innovation,  as  reflected  in  its  Joint  Vision  2010  document  and  other  initiatives  of 
the  era  (Joint  Staff,  1996). 
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6  Strategic  considerations  include  overall  force  size,  the  portfolio  of  capabilities, 
and  how  to  go  about  achieving  various  classes  of  capability. 

7  For  the  basic  character,  see  DoD,  1992.  Although  its  source  is  not  shown,  it  is 
related  to  or  part  of  the  defense  planning  guidance  for  fiscal  years  1994-1999. 

8  Not  much  information  is  readily  available  in  the  public  literature  about  DoD’s 
planning  scenarios.  One  overview  is  quite  useful  in  this  regard  (Allen,  2011),  indi¬ 
cating  the  nature  of  both  single-challenge  scenarios  and  integrated  security  con¬ 
structs  that  postulate  multiple  challenges  overlapping  in  time  (e.g.,  defeat  a  regional 
rogue,  deal  with  a  terrorist  attack  on  the  homeland,  and  deter  other  adversaries).  See 
also  Gates,  2010,  pp.  41-43. 

9  The  2010  QDR  considered  variations  stressing  U.S.  capabilities  in  diverse  ways 
(Gates,  2010,  pp.  41-43).  The  related  analysis  depended  on  quick  and  simple  analy¬ 
sis,  not  campaign  models  or  detailed  scenarios.  This  analysis  benefited  greatly,  how¬ 
ever,  from  the  some  analytic  baselines  on  the  shelf  and  from  the  analytic  exper¬ 
tise  of  those  in  CAPE’s  SAC  who  had  worked  with  the  SSA  campaign  models  and 
scenarios.  When  the  campaign-modeling  was  terminated,  baselines  were  no  longer 
developed,  and  expertise  disappeared  as  well. 

10  See  discussion  and  references  cited  in  Davis,  2014. 

11  Tow  resolution  can  be  illustrated  with  two  examples:  (1)  a  strategic-nuclear  sce¬ 
nario  specifies  the  alert  state  of  U.S.  forces  and  the  type  of  retaliatory  attack  for 
which  it  is  ready;  (2)  a  strategic-mobility  scenario  requires  airlift  capacity  as  mea¬ 
sured  in  millions  of  ton-miles  per  day,  without  specifying  what  is  lifted  or  where  it 
goes.  Such  low-resolution  depictions  should  be  approximately  consistent  with  more- 
detailed  descriptions. 

12  This  assertion  reflects  discussions  with  the  late  Ted  Warner,  ASD  for  Strategy  and 
Resources  in  the  1990s;  other  veterans  of  the  office;  and  personal  observations. 

13  Others  have  argued  for  much  broader  reasons  that  OSD  would  benefit  from 
a  dedicated  decision-support  cell  focused  on  strategic  issues  (Tamb  and  Tachow, 
2006). 

14  An  early  version  appeared  in  a  study  sponsored  by  PA&E  through  the  Defense 
Modeling  and  Simulation  Office,  which  generated  two  reports  (Davis  and  Hen- 
ninger,  2007;  Allen  et  al.,  2007).  The  study  pulled  together  ideas  raised  and  debated 
in  analytic-community  workshops  to  suggest  a  master  plan  for  analysis.  A  version  of 
the  first  report  was  adopted  as  part  of  National  Research  Council,  2006.  That  ver¬ 
sion  appeared  in  Davis,  2014. 

15  Ironically  even  with  the  recent  emphasis  on  operational-level  wargames,  DoD 
still  tends  to  focus  on  state-on-state  wars,  such  as  in  Eastern  Europe  and  the  Pacific 
(e.g.,  Taiwan  Straits,  South  China  Sea).  There  has  been  relatively  little  study  of  ISIL 
and  other  types  or  irregular  or  unconventional  wars. 
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16  A  large  published  volume  summarized  the  initial  work  (Davis  and  Cragin,  2009). 
Work  on  public  support  for  terrorism  was  then  released  (Davis,  Larson,  et  ah,  2012). 
It  included  new  case  studies  testing  validity  of  the  qualitative  models.  The  models 
passed  this  test  in  the  context  of  providing  explanation,  structure,  and  useful  but 
uncertain  diagnosis  and  prescription.  The  models  were  not  intended  to  be  predictive. 

17  The  earliest  version  was  sponsored  by  the  Office  of  Naval  Research  as  part  of 
DoD’s  Human  Social  Culture  Behavior  Modeling  Program  and  by  the  Marine 
Corps  (Davis  and  O’Mahony,  2013).  This  has  led  recently  to  research  on  heteroge¬ 
neous  fusion — i.e.,  fusion  of  different  kinds  of  information  from  diverse  sources  and 
with  such  characteristics  as  being  soft,  ambiguous,  contradictory,  and  even  decep¬ 
tive  (Davis,  Perry,  et  ah,  2016). 

18  Variants  of  the  definition  appear  in  numerous  sources,  such  as  Lempert,  Popper, 
and  Bankes,  2003,  a  report  on  very-long-term  planning. 

19  For  a  review  of  concepts  dating  back  to  the  1980s  and  national-security  applica¬ 
tions  since  the  early  1990s,  see  Davis,  2012.  For  recent  applications  see  also  Lem¬ 
pert,  Warren,  et  ah,  2016. 

20  DoD  has  used  the  term  wargame  to  mean  “human  wargame,”  whereas  others 
have  often  used  the  term  to  include  the  use  of  computer  models. 

21  The  memorandum  was  issued  in  May  2015  (Work,  2015). 

22  See  Beall,  2015,  for  discussion  by  a  senior  executive  on  the  Navy  staff  with  long 
experience  in  both  naval  and  joint  studies  and  wargaming. 

23  A  well-known  book  on  human  wargaming  was  developed  at  the  Center  for  Naval 
Analyses  by  Peter  Perla  (Perla,  1990)  and  has  been  recently  updated  (Curry,  2012). 
Many  analysts  who  see  value  in  wargaming,  including  this  author,  have  been  influ¬ 
enced  over  the  years  by  commercial  contributions,  particularly  by  James  Dunnigan 
(Dunnigan,  2003).  The  Office  of  Net  Assessment  has  used  human  wargaming  since 
its  inception. 

24  One  published  example  dealt  with  Palestine-Israel  issues  (Molander  et  ah,  2009). 
Roger  Molander  was  one  of  the  primary  figures  in  the  Day  After  games,  in  addition 
to  Peter  Wilson  of  RAND. 

25  For  another  discussion  of  how  to  relate  gaming  back  to  analysis,  see  Perla  and 
McGrady,  2009. 

26  The  model-test-model  approach  is  common  in  some  kinds  of  high-technology 
industry,  including  development  of  advanced  missiles.  Since  it  is  not  possible  to 
test  such  systems  exhaustively  in  the  field,  decisions  must  depend  on  believing  the 
model  of  the  system.  Field  tests,  then,  are  designed  to  test  or  provide  information  on 
aspects  on  the  model  that  are  most  in  need  of  empirical  information. 
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27  This  work  is  being  led  under  CAPE  sponsorship  by  Joel  Predd  and  Igor  Mikolic 
Torreira  of  RAND.  The  intent  is  “exploratory  force-structure  analysis.”  No  unclas¬ 
sified  documentation  exists  as  yet. 

28  Most  analysts  in  CAPE  frame  issues,  review  other  analyses,  and  sometimes  do 
back-of-the-envelope  calculations  to  inform  decisions.  Much  of  these  analysts’  time 
is  spent  on  programmatic  issues — defining  the  issues,  leading  program  review  teams, 
dealing  with  cost  estimates  and  budgets,  etc.  Service  analysts  are  much  more  likely 
to  build  and  use  models  to  perform  detailed  studies  (e.g.,  analysis  of  alternatives). 


Note  to  Appendix  A 

1  The  author  confirmed  in  discussions  with  staff  of  the  House  Armed  Services 
Committee  that  the  primary  intent  of  the  study  was  to  assess  the  SSA  activity. 


Notes  to  Appendix  B 

1  Intent  was  reflected  by  the  Principal  Under  Secretary  of  Defense  (Policy),  Ryan 
Henry,  in  an  address  to  the  Military  Operations  Research  Society  in  which  he 
said  that  the  directive  would  “link  DoD-wide  decisions  with  the  defense  strategy 
and  apportion  risk  across  the  range  of  challenge  areas:  traditional,  irregular,  cata¬ 
strophic,  and  disruptive”  (quoted  in  Bexfield  and  Disbrow,  2004,  p.  8). 

2  Little  of  this  occurred,  although  two  such  analytic  baselines  were  developed  by 
U.S.  Pacific  Command. 


Note  to  Appendix  C 

1  The  data  at  the  Institute  for  Defense  Analyses  website  (Institute  for  Defense 
Analyses,  undated)  have  been  mapped  approximately  into  the  same  structure  as  that 
for  RAND. 


Notes  to  Appendix  D 

1  The  Net  Assessment  Office  has  remained  outside  the  fray  of  the  PPBE  and  acqui¬ 
sition  processes,  although  it  contributes  research  and  analysis.  It  has  also  tended  not 
to  be  especially  analytic  in  the  sense  of  that  term  used  in  SSA  circles:  The  office  has 
often  emphasized  softer  methods,  such  as  studying  technological  trends,  human 
wargaming,  applying  the  concepts  of  strategic  competition,  and  thinking  creatively 
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about  possible  future  scenarios  (Krepinevich,  2010).  Analytical  models  have  played 
only  a  small  role  in  its  work.  There  were  exceptions  in  the  1980s,  but  the  office  con¬ 
cluded  subsequently  that  the  other  methods  were  more  productive  and  efficient  for 
its  purposes. 

2  An  example  is  the  analysis  of  alternatives  for  aerial  refueling  (Kennedy,  2006). 


Abbreviations 


ASD 

Assistant  Secretary  of  Defense 

ASSP 

Analytic  Support  for  Strategic  Planning 
(proposed) 

CAPE 

Cost  Assessment  and  Program  Evaluation 

CONOPS 

concepts  of  operations 

DoD 

Department  of  Defense 

FARness 

flexibility,  adaptiveness,  and  robustness 

FFRDC 

federally  funded  research  and  development 
center 

JICM 

Joint  Integrated  Contingency  Model 

M&S 

modeling  and  simulation 

MS&A 

modeling,  simulation,  and  analysis 

MSFD 

Multi-Service  Force  Deployment 

NATO 

North  Atlantic  Treaty  Organization 

OSD 

Office  of  the  Secretary  of  Defense 

OUSDP 

Office  of  the  Under  Secretary  of  Defense  for 
Policy 

PA&E 

Office  of  Program  Analysis  and  Evaluation 
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PPBE 

planning,  programming,  budgeting,  and 
executing 

QDR 

Quadrennial  Defense  Review 

SAC 

Simulation  and  Analysis  Center 

SSA 

support  for  strategic  analysis 

USD(AT&L) 

Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics 

WSEG 

Weapon  Systems  Evaluation  Group 
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The  U.S.  Department  of  Defense  (DoD)  has  long  had  broad  capabilities  for  joint 
analysis,  whether  for  planning  future  forces  or  supporting  field  commanders 
during  current  challenges.  DoD  is  envied  by  other  government  agencies  for  its 
analytic  capabilities  and  processes.  Nonetheless,  issues  have  arisen.  This  report 
stems  from  a  congressional  request  for  an  independent  report  about  DoD's 
capabilities  for  joint  analysis  and  ways  to  improve  them.  Congressional  concerns 
largely  involved  the  activity  called  support  for  strategic  analysis  (SSA).  In  2011, 
the  director  of  the  Office  of  the  Secretary  of  Defense's  (OSD's)  Cost  Assessment 
and  Program  Evaluation  (CAPE)  disbanded  campaign-modeling  and  reduced 
CAPE's  participation  in  the  SSA  activity,  weakening  SSA  significantly.  This  report 
is  largely  about  SSA  and  how  and  whether  to  revise  it. 

The  author  drew  on  previous  research  and  conducted  new  interviewees  with 
present  and  past  DoD  officials  and  working  analysts.  The  concerns  expressed 
by  interviewees  fell  into  in  two  groups:  those  seeing  great  value  in  various  aspects 
of  the  SSA,  including  the  campaign-modeling  and  analysis  for  which  capability 
is  seen  to  have  plummeted,  and  those  giving  low  marks  to  the  SSA  processes  and 
some  products.  The  research  concluded  that,  rather  than  making  on-the-margin 
tweaks  to  SSA,  DoD  should  make  fundamental  revisions  to  the  overall  planning 
construct  to  which  SSA  contributes.  Doing  so  will  lead  naturally  to  changes  in 
functions,  organization,  process,  methods  and  tools,  and  staffing.  The  author 
recommends  creating  a  new  activity  to  assist  in  mid-  and  longer-term  planning, 
called  analytic  support  for  strategic  planning. 
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